August 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Sabreen Alikhan, Maryam Hassan, Sula Nakanyika-Mahoney, Moses Mbawo, and Lindsey Carpenter Toomey on behalf of Social Impact, Inc. STRENGTHENING LOCAL CAPACITY TO COMBAT HIV/AIDS IN ZAMBIA End of Project Performance Evaluation i Cover Photo: Walkway to Partner Organization Harvest Help Zambia in Siavonga (June 2013). ii STRENGTHENING LOCAL CAPACITY TO COMBAT HIV/AIDS IN ZAMBIA: END OF PROJECT PERFORMANCE EVALUATION EXPERIENCE FROM A PARTNERSHIP WITH LOCAL ORGANIZATIONS AND INDIVIDUALS WORKING COOPERATIVELY TO STRENGTHEN THE HIV/AIDS RESPONSE IN THE COMMUNITY August 2013 RAN-I-00-09-00019/AID-611-TO-13-00004 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii ACKNOWLEDGMENTS This evaluation could not have been possible without the contributions of dedicated USAID, implementing partner, and beneficiary organization staff, whose inputs were instrumental to the evaluation team’s efforts. Firstly, the team extends its gratitude to Monitoring and Evaluation Specialist and Contracting Officer’s Representative, Patricia Sitimela, for her tremendous support and guidance from the evaluation’s inception to its close-out. The team would also like to thank Dr. Cynthia Bowa, HIV/AIDS Multisectoral Deputy Team Leader, Ky Lam, HIV/AIDS Multisectoral Team Leader, and Ngaitila Phiri, Alternate Contracting Officer’s Representative, for taking the time to provide critical information to the team, as well as thoughtful feedback on deliverables. Thanks are also due to USAID Mission Director, Dr. Susan Brems, Deputy Mission Director, Ryan Washburn, Emma Sitambuli, Program Development Specialist, and affiliated Mission staff, as well as the USAID/Washington Bureau for Policy, Planning, and Learning, Office of Evaluation and Research (PPL/LER), for the direction provided throughout this evaluation. The team also wishes to acknowledge the tremendous support it received from staff of the implementing partner, FHI 360, and its sub-partner, Management Systems International (MSI). Specifically, the team expresses its deep appreciation for the generosity of LPCB Chief of Party, Cara Endyke-Doran, and LPCB Deputy Chief of Party, Francis Johnston, in their willingness to welcome the team, provide critical input to numerous inquiries, supply project data, and facilitate introductions to partner organizations. The team also acknowledges FHI 360 and MSI home office staff for their timely support in responding to the team’s additional requests. The team would like to convey its bottomless appreciation for the staff of each local partner organization sampled for this evaluation. The team is especially appreciative of the flexibility of partner organization staff to accommodate the evaluators into their busy schedules. These staff consistently greeted the team with warmth and enthusiasm, making available time and resources to provide the information upon which this evaluation is based. Last, but not least, the team would like to thank Dennis Wood and Alexandra Ernst for their home office technical and administrative support throughout the duration of this work. Special thanks are also extended to the team’s local logistics coordinator, Natalie Mufalo, whose support contributed a critical component of this evaluation effort. iv CONTENTS ACKNOWLEDGMENTS ............................................................................................................. iii ACRONYMS .................................................................................................................................. 1 MAP OF PROJECT AREA................................................................................................................ 2 EXECUTIVE SUMMARY.............................................................................................................. 3 EVALUATION PURPOSE & EVALUATION QUESTIONS .................................................... 8 PROJECT BACKGROUND ........................................................................................................ 10 EVALUATION METHODS & LIMITATIONS ......................................................................... 12 FINDINGS & CONCLUSIONS .............................................................................................. 1817 1: ACHIEVEMENT OF LPCB OBJECTIVES ............................................................................... 1817 Findings ..........................................................................................................................................................................................1817 Conclusions..................................................................................................................................................................................2625 Lessons Learned ..........................................................................................................................................................................2726 2: LPCB DESIGN, IMPLEMENTATION, AND MANAGEMENT ................................................ 2827 Findings ..........................................................................................................................................................................................2827 Conclusions ..................................................................................................................................................................................3231 Lessons Learned ..........................................................................................................................................................................3332 3: BENEFICIARY ANALYSIS OF OUTCOMES.......................................................................... 3433 Findings ..........................................................................................................................................................................................3433 Conclusions ..................................................................................................................................................................................4342 Lessons Learned ..........................................................................................................................................................................4443 4: QUALITY HIV/AIDS SERVICE DELIVERY............................................................................. 4544 Findings ..........................................................................................................................................................................................4544 Conclusions ..................................................................................................................................................................................5049 Lessons Learned ..........................................................................................................................................................................5049 5: SUSTAINABILITY OF RESULTS............................................................................................... 51 Findings ...............................................................................................................................................................................................51 Conclusions .......................................................................................................................................................................................52 Lessons Learned...............................................................................................................................................................................53 RECOMMENDATIONS .............................................................................................................. 54 v ANNEXES Annex I: Evaluation Statement of Work Annex II: Evaluation Methods and Limitations Annex III: Data Collection Instruments Annex IV: Sources of Information Annex V: Disclosure of Any Conflicts of Interest Annex VI: LPCB Indicators and Results against Targets TABLES & FIGURES Table 1. Data Collection and Analysis Matrix Limitations Table 2. Contribution Analysis Evaluation Approach Table 3. Partner Organization Sampling Matrix Table 4. Key Project Inputs and Linked Outcomes by Cohort Table 5. Key Project Inputs and Linked Outcomes by Province Table 6. Select PO Measures: Sampled Women-led POs vs. Other Table 7. Aggregate P8.1 Reporting for Select POs Figure 1. PEPFAR Indicator Results Reported by LPCB Beneficiary Organizations Figure 2. Gender Disaggregated PEPFAR Indicator Results Reported by LPCB Beneficiary Figure 3. LOP Aggregate PEPFAR Results against Targets Figure 4. Reported Changes in PO Financial Status Post-LPCB Figure 5. Reported Changes in Service Delivery among Sampled POs 1 ACRONYMS AB Abstinence and/or Being Faithful AIDS Acquired Immune Deficiency Syndrome CL Capacity Leader Organization CB Capacity Building CBO Community-Based Organization FBO Faith-Based Organization FGD Focus Group Discussion FY Fiscal Year HCW Health Care Worker HIV Human Immunodeficiency Virus IDF Institutional Development Framework IGA Income-Generating Activity IODF Institute of Organization Development Facilitators KII Key Informant Interview KQ Key (Evaluation) Question LPCB Local Partners Capacity Building LOP Life of Project M&E Monitoring and Evaluation MARP Most At-Risk Population NAC National AIDS Council NGO Non-Government Organization OD Organization Development ODF Organization Development Facilitator PEPFAR President’s Emergency Plan for AIDS Relief PLHIV People Living with HIV PO Partner Organization PwP Prevention with Positives SD Service Delivery SOW Statement of Work TA Technical Assistance USAID U.S. Agency for International Development ZPI Zambia-led Prevention Initiative 2 MAP OF PROJECT AREA [Type a quote from the document or the summary of an interesting point. You can position the text box anywhere in the document. Use the Drawing Tools tab to change the formatting of the pull quote text box.] < 5 PROJECT SITES 5 – 10 PROJECT SITES > 10 PROJECT SITES PROVINCES VISITED BY EVALUTION TEAM NORTHWESTERN WESTERN SOUTHERN LUSAKA CENTRAL COPPERBELT LUAPULA NORTHERN MUCHINGA EASTERN 3 EXECUTIVE SUMMARY This end-of-project performance evaluation was conducted by Social Impact, Inc. between May and July 2013. The evaluation team comprised of two international and two local specialists, employed a mix of qualitative and quantitative methods to assess the effectiveness of USAID/Zambia’s flagship capacity building initiative, Local Partners Capacity Building (LPCB) project. The team’s resultant findings, conclusions, lessons learned, and recommendations are presented below. EVALUATION PURPOSE AND QUESTIONS The purpose of this evaluation is to assess the effectiveness and efficiency of USAID/Zambia’s Local Partners Capacity Building (LPCB) project in order to inform future programming in strengthening local organizational capacity. This end-of-project evaluation took place between May and July 2013, as the five-year-long project activities came to a close. The evaluation sought to address the following prescribed key questions (KQs): 1. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? 2. To what extent were the project design, implementation, and management effective and why? 3. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? 4. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? 5. What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? PROJECT BACKGROUND A five-year initiative, LPCB was launched in 2008 by the Academy for Educational Development (AED), now FHI 360, to support Zambia’s local organizations in effectively absorbing the influx of donor funding allocated to combating HIV/AIDS in Zambia. As USAID/Zambia’s leading capacity building project, LPCB sought to increase the technical and institutional capacity of “partner organizations,” an umbrella term that covers non-governmental organizations (NGOs), faith-based organizations (FBOs), and community￾based organization organizations (CBOs) to both effectively account for and program increased donor funding. The objectives of the project were to: assess organizations’ capacity using a self-assessment tool (Institutional Development Framework); strengthen technical and institutional capacity through training and direct technical assistance (through local Zambian Capacity Leader Organizations or individual Organization Development Facilitators); provide funding opportunities to local partners demonstrating increased capacity; assist organizations in documenting impact and disseminating evidence-based HIV/AIDS services; and sustaining in-country capacity to provide technical support for organizational strengthening. LPCB was implemented throughout all ten provinces of Zambia. Local partners participating in LPCB 4 were grouped together into five separate, overlapping cohorts comprised of approximately 20 organizations each. Each cohort was composed of organizations from different provinces and of varying sizes and levels of capacity. Following training in a suite of core technical and organizational competencies, organizations were provided the opportunity to apply for a grant from LPCB. Successful grant recipients were given an average of one year in which to implement grant-funded activities, resulting in an overall term of engagement of one to two years for each participating partner organization. Upon conclusion of LPCB engagement, organizations were rated as having graduated (exceeded grant objectives), completed (met grant objectives, but some capacity thresholds unmet), or participated (unsuccessful grant attainment and/or completion of trainings). EVALUATION DESIGN, METHODS AND LIMITATIONS Due to the lack of valid control groups of local organizations, a non-experimental performance evaluation design was employed for this evaluation. This evaluation was intended to measure quantitative and qualitative changes that have occurred in subject organizations, their staff, and broader communities. To the extent possible, the evaluation team made before and after comparisons using organizations’ self￾reported capacity scores and progress toward established targets; however, the team acknowledges the effect that subjective measures have on the internal validity of data. Therefore, triangulation of data from multiple sources was used to corroborate self-reported data. From the universe of 107 beneficiary organizations, the evaluation team purposively sampled 35 Partner Organizations (POs) representing a range of PO size, performance level, and geographic location, as well as all four Capacity Leader Organizations (CLs). Semi-structured questionnaires were used to elicit qualitative and quantitative inputs from sampled organizations. The team also conducted key informant interviews with representatives of USAID, LPCB, Institute of Organization Development Facilitators (IODF), National AIDS Council (NAC), and Zambia-led Prevention Initiative (ZPI) using structured questionnaires customized for each stakeholder. The evaluation team visited five provinces in order to conduct qualitative data collection; however, due to time and logistical constraints, only one province designated by USAID/Zambia as “rural” could be visited. The team was requested to sample six Partner Organizations from each of five cohorts. Despite many attempts, a number of scheduling conflicts and declines for interview resulted in the team only sampling three POs from Cohort 3. To compensate for this, organizations from other cohorts were purposively oversampled. In the absence of a random sample selection, the generalizability of findings from qualitative interviews is limited in scope. Additionally, qualitative interviews are inherently subject to biases. The evaluation team used best practices in evaluation to minimize bias and subjectivity to enhance the rigor of the evaluation results. Due to the inability to access a complete dataset on annual PEPFAR results disaggregated by partner organization, the evaluation team was unable to perform certain analyses. However, before-and-after, chi square, and regression analyses were performed on available data, and where applicable, limitations to the generalizability of the conclusions of such analyses are explicitly stated throughout the report. Due to the ex post nature of the evaluation and the lack of valid baseline data or control group, beneficiary impacts were not able to be attributed directly to project inputs. 5 KEY FINDINGS AND CONCLUSIONS 1. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? Findings Of the 35 partner organizations interviewed, 86% Partner Organizations across all cohorts and provinces reported that the Institution Development Framework tool was effective in gauging organizational capacity and identifying areas for improvement. The project far exceeded its targets for number of health care workers successfully trained (648% target achieved), number of POs offered networking opportunities (206% target achieved), number of POs that received grants through LPCB (180% target achieved), and number of grant recipients that achieved at least 75% of stated objectives (159% target achieved). The number of Capacity Leader Organizations (CLs) enrolled in LPCB and the number of individuals trained as independent Organization Development Facilitators (ODFs) fell below Life of Project targets. Though the numbers reported decline over the life of the project, LPCB exceeded its targets for the main PEPFAR prevention indicator (P 8.1) and its subset indicators. Partner organizations were most successful in their delivery of Prevention through Abstinence/Being Faithful messaging, followed by Prevention with Positives (PwP) and Prevention among most at-risk populations (MARPs). Conclusions With the exception of two measures, the project achieved its target indicators and planned activities. Overarching project objectives of organizational assessment, strengthened institutional capacity, funding, and documenting impact were largely met by the project activities. Partner Organizations’ short term of engagement with LPCB makes it difficult to determine the success of the project’s sustainability objective. The appearance of reduced success against targets over the life of the project suggests the effectiveness of LPCB in strengthening technical capacity of POs; by conforming to evidence￾based strategies focused on individual and small group-level interventions over mass campaigns, POs generally demonstrated increased technical capacity to deliver quality services. 2. To what extent were the project design, implementation, and management effective and why? Findings Smaller organizations reported LPCB’s “one size fits all” approach to training inhibited their ability to absorb, and ultimately apply, core competencies in institutional and technical capacity as compared to their larger, higher-capacity counterparts. The establishment and implementation of graduation plans contributed to overall project results by incentivizing success (graduated POs were recommended to external donors) and providing an additional measure by which to categorize an organization’s growth as a result of LPCB. The Institutional Development Framework (IDF), a self-assessment tool used by partner organizations to rate their own capacity, was found to be one of the most highly valued components of LPCB; 80% of POs report having adapted the IDF for their own use with the intention of assessing organizational capacity on a regular basis. 6 Conclusions Smaller POs were disadvantaged by the design of cohort groupings. LPCB demonstrated responsiveness to lessons learned over the life of the project and effectively addressed project gaps identified by the mid-term assessment, such as the need for PO graduation plans. While the IDF is an effective tool in a qualitative sense, it is a poor determinant of organizational success due to its inherent subjectivity. 3. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? Findings Of the 35 POs sampled for interviews, 86% reported an increase in general operational capacity due to LPCB. The most frequently cited effect of LPCB was the establishment of systems; POs across all cohorts and provinces, regardless of graduation status, cited the development and adherence to proper systems as one of the most significant impacts of LPCB. Of the 35 POs sampled, 54% reported an increase in management capacity during and after LPCB engagement. POs from rural areas were more likely to rate the impact of governance training as having played a critical role in its overall capacity development, citing the management challenges faced prior to LPCB. The most frequently cited technical skills gained among Cohorts 1 and 2 included the knowledge and ability to target messaging to specific populations, whereas POs from Cohorts 3, 4 and 5 were more likely to cite a shift from mass prevention campaigns to small group-focused interventions. Of the POs sampled, 46% reported an increase in the number of funding sources since disengaging from LPCB. Twenty-three percent reported an increase in both number of sources and overall level of funding, while only nine percent reported an increase in funding level alone. Conclusions LPCB considerably strengthened institutional and technical capacity of beneficiary organizations. For smaller POs, the largest impact was the creation of systematized operations and standards. For larger POs, existing systems were sharpened and refined. POs reported that increased technical capacity allowed them to improve the quality of HIV/AIDS service delivery. LPCB created demand for additional technical capacity among beneficiaries. All organizations experienced improved financial viability as a result of LPCB, even if only provided with training. Sampled women-led organizations in particular have benefitted from income-generating activities. 4. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? Findings Seventy-one percent of POs interviewed stated that the quality of their organization’s service delivery had been positively affected by participation in LPCB. Fifty-three percent POs surveyed reported in an increase in the quantity of their HIV/AIDS service delivery, primarily as a function of expanding the scope of their target population in 7 response to improved technical skill sets and heightened awareness of evidence-based communication strategies. LPCB assumed a logical progression from improved organizational capacity to increased quantity and quality of HIV/AIDS services without articulating the predicted linkages in a development hypothesis. LPCB’s disparate goals – a dual focus on quantity and quality – created a mismatch between the project’s indicators, (focused on quantity), and the project’s perceived and stated goals, (focused on quality). As a result, several POs reported a deliberate decision to prioritize quantity, even at the expense of reaching fewer beneficiaries. Conclusions The project successfully increased POs capacity to deliver higher quality HIV/AIDS services, while increasing the actual delivery of services to a lesser, and indirect, extent. The tenuous link between the LPCB goal of increased and improved service delivery and its primarily organizational development-focused inputs created confusion among POs about the purpose of the project. The link between OD and SD exists, but LPCB’s lack of strategic planning in PMP development at the project’s inception diluted its effects and left the project’s potential to establish a valuable theory of change unfulfilled. 5. What are the prospects for the sustainability of the capacity building results for the local partner organizations and institution strengthening providers? Findings Of the 35 POs sampled, 60% reported that the one project component enabling organizations to remain sustainable in the future was training. Thirty-four percent of sampled POs reported the establishment of systems as critical to organizational sustainability. LPCB served as a catalyst in the spurring the development of spinoff collaborations among POs wishing to broaden their collective community impacts, such as the Civil Society Framework for Responding to HIV, TB, and Malaria in Zambia (CSF) and the Local NGO Directors’ Forum. World Vision, a longtime actor in the Zambian HIV/AIDS arena, was reported to have approached LPCB with the purpose of adopting a similar model for its own capacity building project. Membership of the Institute of Organization Development Facilitators (IODF) has grown beyond LPCB-affiliated Organization Development Facilitators to include additional individuals and organizational stakeholders in local capacity building. Conclusions Best prospects for PO sustainability include networking opportunities and the development of demonstrable systems and skills, which hinge on the successful provision of technical refresher trainings. Private sector donors have begun to imitate LPCB’s design and approach in response to the visibility of LPCB achievements. LPCB has succeeded in building a cadre of OD professionals through the IODF, but its sustainability is challenged by a general shortage of funding opportunities. ________________________________________ 8 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this evaluation is to assess the effectiveness and efficiency of USAID/Zambia’s Local Partners Capacity Building (LPCB) project in order to inform future programming in strengthening local organizational capacity. This end-of-project evaluation took place between May and July 2013, as the five-year-long project activities came to a close. The overall objectives of the evaluation are: 1. To assess the extent to which project objectives, targets, outputs and expected results were achieved and or exceeded (performance) in accordance with the LPCB Performance Monitoring and Evaluation Plan. 2. To assess the effectiveness and potential sustainability of project activities and capacity building model(s) and approaches on the institutional and technical capacity of local organizations and subsequent expansion of quality community-based HIV/AIDS services. 3. To inform USAID/Zambia of lessons learned and best practices for replicating and scaling up local capacity development models and identify promising and high performing local partners (replicability). It is anticipated that the evaluation will be useful to multiple audiences and stakeholders including inter alia: USAID/Zambia: To identify effective practices and areas for improvement and to inform the design and implementation of future interventions in building local organizational capacity in Zambia. To identify promising and competent local partner organizations among the POs and institution strengthening providers that could become viable direct local partners of USAID. USAID/Global Health Bureau, PEPFAR: To demonstrate the effectiveness and efficiency of engaging and strengthening the capacity of local organizations in expanding the delivery of quality HIV/AIDS services. USAID/Policy Planning and Learning Bureau: To suggest a model for evaluating capacity building programs. Government of the Republic of Zambia, and other donors: To demonstrate the effectiveness of strengthening local organizational capacity in expanding the delivery of HIV/AIDS services. LPCB, POs, and Institutional strengthening providers: To learn their strengths and weaknesses and adjust their technical approaches for future projects accordingly. USAID/Zambia will use findings from the evaluation to inform the design of future projects and publications on capacity building interventions for local organizations. The report will be disseminated widely among relevant stakeholders and project beneficiaries as well as submitted to the Development Exchange Clearing House (DEC). 9 EVALUATION QUESTIONS The evaluation addresses a constellation of interrelated KQs: 1. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? a. Assess whether the activity managed to achieve the planned results focusing on quality/quantity of outputs for this activity. Assess the factors that facilitated or inhibited the achievement of these results. 2. To what extent were the project design, implementation, and management effective and why? a. Assess the best practices and lessons learned during each of the phases. Indicate any changes that occurred during implementation of this activity, both the external environment and or internal to the activity, in the evaluation report especially where they may have had a bearing on activity outputs and outcomes. Assess the effectiveness of the tools used to track changes in organizational capacity and whether these were good predictors of organizational success. Assess the effectiveness and efficiency of the institutional strengthening providers and areas of comparative advantage. 3. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? a. Assess whether these changes or outcomes were comparable across cohorts and also across the provinces. Assess whether the successes accrued equally to men or women-led organizations. Compare changes that occurred during the period when the organizations were receiving LPCB support and those that happened after their support had ended. Indicate other concurrent organizational strengthening support that the organizations received with the LPCB activity especially where this may have had a bearing on organizational success. Assess the potential of the organizations moving on to become direct recipients of USAID funding. 4. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? 5. What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? 10 PROJECT BACKGROUND Zambia’s struggle to deal with its HIV epidemic remains one of the critical constraints to the country’s economic development. The United States and other donors have significantly increased the level of aid funding devoted to dealing with the HIV/AIDS epidemic in Zambia and elsewhere. The largest global health initiative in United States history, the President’s Emergency Plan for AIDS Relief (PEPFAR), set ambitious goals and targets for HIV prevention, care, and systems strengthening. To help meet these goals, USAID/Zambia recognized the need for rapid and holistic strengthening of the technical and organizational capacity of local groups to improve the ability of existing Zambian organizations to respond to HIV-related challenges with high-quality services. The Local Partners Capacity Building (LPCB) project was developed in response to the need to fill capacity gaps in local service providers offering HIV prevention services. A five-year initiative, LPCB was launched in 2008 by the Academy for Educational Development (AED), now FHI 360, to support Zambia’s local organizations in effectively absorbing the influx of donor funding allocated to combating HIV/AIDS in Zambia. As USAID/Zambia’s leading capacity building project, LPCB sought to increase the technical and institutional capacity of “partner organizations,” an umbrella term that covers non-governmental organizations (NGOs), faith-based organizations (FBOs), and community￾based organization organizations (CBOs) to both effectively account for and program increased donor funding. The objectives of the project were as follows: 1. Assess: Assess Zambian NGOs, FBOs, and CBOs that currently are United States Government (USG) partners or sub-partners, on their organizational, financial and technical capacity with regards to managing, implementing, results reporting and scale-up of HIV/AIDS programs in prevention, care and treatment. 2. Strengthen technical capacity: Strengthen the technical capacity of local NGOs, CBOs, FBOs, and networks so that they can expand and improve the quality of their HIV/AIDS services and activities, link to service delivery networks, and advocate effectively for legislative and policy change. 3. Strengthen institutional capacity: Support promising PEPFAR-supported local partners and sub-partners with the technical assistance and financial resources to put effective management and financial systems in place, develop and implement business plans, and manage organizational change. 4. Fund: Provide PEPFAR funding opportunities to local USG partners and subpartners that have achieved the requisite organizational strengthening for the scale-up of HIV/AIDS activities and services, using an umbrella award model. 5. Document: Assist local organizations to document impact and disseminate evidenced-based innovations, best practices and lessons learned. 6. Sustain: Create in-country capacity to provide technical support for organizational strengthening. 11 LPCB was designed to contribute to the USAID/Zambia 2004-2010 strategic objective of Reduced Impact of HIV/AIDS through a Multisectoral Response, and associated intermediate results, as well as Zambia’s annual PEPFAR prevention, care and systems strengthening targets. The project’s capacity building models resonate USAID and PEPFAR’s emphasis on country ownership. The LPCB logic model, as gleaned from project documents and the LPCB Associate Cooperative Agreement, is as follows: LPCB was implemented throughout all ten provinces of Zambia. Local partners participating in LPCB were grouped together into five separate, overlapping cohorts comprised of approximately 20 organizations each. Each cohort was composed of organizations from different provinces and of varying sizes and levels of capacity. Following training in a suite of core technical and organizational competencies, organizations were provided the opportunity to apply for a grant from LPCB. Successful grant recipients were given an average of one year in which to implement grant-funded activities, resulting in an overall term of engagement of one to two years for each participating partner organization. Upon conclusion of LPCB engagement, organizations were rated by project staff as having graduated (exceeded grant objectives), completed (met grant objectives, but some capacity thresholds unmet), or participated (unsuccessful grant attainment and/or completion of trainings). SUSTAIN Country-led ownership of HIV/AIDS response INCREASE Quantity and quality of HIV/AIDS service delivery BUILD Technical and institutional capacity of local partners organizations and individuals PROVIDE Technical assistance and funding opportunities for institutional capacity development Cohort 1 Cohort 2 Cohort 3 Cohort 4 Cohort 5 Year 1 Year 2 Year 3 Year 4 Year 5 12 EVALUATION METHODS & LIMITATIONS A detailed evaluation methodology, including strategies and limitations, may be found in Annex II. Per USAID guidelines, a performance evaluation focuses on descriptive and normative questions, such as what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period), how it is being implemented, how it is perceived and valued, whether expected results are occurring; and other questions that are pertinent to program, design, management and operational decision making. Due to the lack of valid control groups of local organizations, a non-experimental performance evaluation design was employed for this evaluation. This evaluation was intended to measure quantitative and qualitative changes that have occurred in subject organizations, their staff, and broader communities. To the extent possible, the evaluation team made before and after comparisons using organizations’ self-reported capacity scores and progress toward established targets; however, the team acknowledges the effect that subjective measures have on the internal validity of data. Therefore, triangulation of data from multiple sources was used to corroborate self-reported data. The Data Collection and Analysis Matrix in Table 1 below is organized around each of the evaluation’s 5 Key Questions (KQs), and provides a description of data collection methods used. To the extent possible, the evaluation team applied a quantitative analysis to answer KQs related to the performance of the project against targets. Data gathered over the life of the project as part of PO M&E, including capacity self-assessment scores, was extracted and analyzed for trends and objective themes that reveal the extent to which PO capacity has been developed and sustained. Chi square and regression analyses were conducted using PO graduation status and financial data, which attempted to identify relationships between levels of support provided and extent of organizational capacity built. Data for these quantitative analyses were derived from the LPCB database and documents provided by project staff. Interlinked methodological components of the LPCB performance evaluation was carried out simultaneously, allowing for the continued triangulation of qualitative and quantitative findings throughout the data collection period. Triangulated findings formed the basis for the synthesis of project results and formulation of cohesive, policy-relevant conclusions and recommendations. In order to explore attribution of impacts to LPCB activities, the team utilized the approach of contribution analysis. This type of analysis, used to verify a program’s theory of change while investigating alternative explanations contributing to observed outcomes, is particularly valuable for evaluation of non-experimental programs. Table 2 below outlines the team’s tacit approach to identifying and consolidating all potential contributions to impact due to LPCB activities and factors unrelated to the program. 13 Table 1. Data Collection and Analysis Matrix Limitations Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data analysis 1. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? Comparative/ Analytic Document review Data Abstraction LPCB M&E database LPCB Work Plans, progress reports N/A Compare observed and reported outputs and outcomes with indicator targets (indicators include PEPFAR as well as LPCB indicators Interviews to understand challenges in meeting targets and revisions to targets. Semi-structured interviews Key informants from beneficiary organizations and local partners Purposive 2. To what extent were the project design, implementation, and management effective and why? Comparative/ Analytic Document Review Data Abstraction Project financial statements LPCB M&E database N/A Analyze relationship of activity levels to output measures; to the extent possible, compare LPCB financial input to project results. Content analysis of interviews data to detect key themes related to implementation and management effectiveness Semi-structured Interviews Key informants from POs Purposive 3. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? Comparative/ Analytic Document review Data Abstraction Regression Analysis Data abstraction from LPCB M&E database PO internal capacity assessment score data N/A (Targeted M&E information on LPCB partners) Analyze relationship of support received to measures of capacity Use qualitative methods followed by content analysis of interviews to uncover themes in tangible capacity development of LPCB beneficiaries. Semi-structured interviews Key informants among LPCB beneficiaries Purposive 4. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? Comparative/ Analytic Document review Regression Analysis LPCB M&E database Results against PEPFAR indicators N/A (Targeted M&E information on LPCB partners) Use quantitative analysis to determine the effect of LPCB on service delivery indicators and outputs at the PO level Content analysis of key informant and beneficiary interview data. Triangulation with observations from field visits to project sites, quantitative and qualitative beneficiary analyses. Semi-structured interviews Stakeholders, key informants, beneficiaries Purposive 5. What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? Analytic Semi-structured interviews Stakeholders, key informants, management and service delivery staff of beneficiary organizations Purposive Content analysis of key informant interviews to discover likelihood of sustainability of project results. 14 Table 2. Contribution Analysis Evaluation Approach KEY LPCB ACTIVITY MECHANISMS ASSUMPTIONS INTENDED OUTCOMES INTENDED IMPACT ALTERNATIVE EXPLANATIONS Training in technical and institutional core competencies Sensitization in areas related to technical service delivery, organization and financial management will build capacity to establish and conform to institutional systems and standards Organizations recognize capacity gaps Training adequate to address demand for capacity and varying organizational needs Trained personnel share materials and knowledge with other staff Trained personnel remain in organization Increased PO technical and institutional capacity Increase quality HIV/AIDS service delivery Knowledge obtained from other sources (e.g. other projects with capacity building components) induce capacity growth Capacity growth affected more directly by higher-level decisions of Board of Directors Technical assistance through CL or ODF Support provided through organizational mentoring will lead to organizations establishing and conforming to institutional systems and standards Quality of support adequate to address organizational capacity gaps Organizations recognize value of conforming to institutional and technical standards Support aligned with individual organization mission and objectives Increased PO technical and institutional capacity Increase quality HIV/AIDS service delivery Support obtained from other capacity building initiatives induce capacity growth Structure of organizational systems and procedures influenced by external donor demands Grant provision LPCB grant provides opportunity for organization to apply proposal development skills Grant implementation provides opportunity to apply technical HIV/AIDS service delivery skills Grant implementation raises organizational capacity to be financially accountable and attain additional funding from external sources Sound M&E and financial reporting systems have been established Organizations meet minimum capacity threshold to sustain grant activities and financial accountability Organizations leverage grant opportunities to seize additional opportunities for funding and capacity growth beyond LPCB Adequate external funding opportunities exist to support increased demand Organizations apply technical skills to raise quality of HIV/AIDS services Organizations possess capacity to verify quality of HIV/AIDS services provided Increased technical capacity leads to organizations increasing target populations Increased PO technical and institutional capacity Increased PO financial viability Increase quality HIV/AIDS service delivery Decision to seek funding opportunities or increase quantity of services affected by factors unrelated to LPCB (e.g. individual PO mission, Board of Directors, etc.) HIV/AIDS service delivery affected by fluctuations in organization-specific target populations Service delivery affected by shifts in organizations’ mission/focus independent of LPCB inputs 15 Data Collection From the universe of 107 beneficiary organizations, the evaluation team purposively sampled 35 Partner Organizations (POs) representing a range of PO size, performance level, and geographic location, as well as all four Capacity Leader Organizations (CLs). Semi-structured questionnaires were used to elicit qualitative and quantitative inputs from sampled organizations. The team also conducted key informant interviews with representatives of USAID, LPCB, Institute of Organization Development Facilitators (IODF), National AIDS Council (NAC), and Zambia-led Prevention Initiative (ZPI) using structured questionnaires customized for each stakeholder. Given the non-experimental nature of the evaluation’s design, it cannot be assumed that selected organizations would be fully representative of the universe of project beneficiaries. Thus, the team endeavored to include as representative as possible a cross section of the project’s universe of beneficiaries by employing a purposive sampling strategy based on characteristics of priority to the focus of this evaluation, including: - PO Type: Non-governmental organizations (NGOs), faith-based organizations (FBOs), and community based organizations (CBOs) - Organization Focus and Mission, such as preventative HIV/AIDS services, community outreach; advocacy, home based care, skills training, etc. - Target Population, to ensure representation of POs targeting most at-risk populations (MARPs), orphans and vulnerable children (OVC), adolescents, and families - PO Characteristics such as organizational size, gender of executive - Graduation Status, to ensure representation of both high- and low-performing POs - Geography: Represent, as time and logistics permit, POs in focal areas of LPCB implementation across urban and rural regions Based on the above selection criteria, a total of 35 Partner Organizations and 4 Capacity Leader Organizations were selected for site visits in the following provinces: Central, Copperbelt, Eastern, Lusaka, and Southern. Of the Partner Organizations sampled, 16 received the LPCB status of “graduated” at the end of project engagement; 11 “completed”, 6 “participated”, and 2 were dropouts. For the purposes of this evaluation, the following definitions apply for descriptions of POs used throughout the report: - Rural: Situated in Eastern Province, Choma, or Siavonga (located > 200 kilometers from nearest major city) - Small: PO core size < 6 staff 16 Table 3. Partner Organization Sampling Matrix COHORT # POS ENROLLED LPCB # POS SAMPLED POS VISITED 1 18 8 - AATAZ - KCDA - CTYA - Kwenuha Women’s Association - YDO - New Masala Theatre Group - Dambwa Christian Care Centre - CODEP 2 23 10 - Bridge of Hope - Chisomo Home Based Care - Katete DWA - CINDI Katete - CBTO - RICAP - Mboole Rural Development Initiative - ZAVCODA - Ndola Catholic Diocese - Flame Community Based Organization 3 19 3 - ZCCP - Roan Youth Development - Restless Development 4 22 8 - Judith Chikonde Foundation - Family Health Trust - Community Health Mobilization - Treatment Advocacy and Literacy Campaign - HIV and AIDS Prevention Network - Afya Mzuri - Girl Guide Association of Zambia - Expanded Church Response 5 21 6 - Harvest Help Zambia - Eastern Province DWA - NZP+ Petauke - Chadiza DWA - Lifeline Zambia - Fleet of Hope Limitations The evaluation team visited five provinces in order to conduct qualitative data collection; however, due to time and logistical constraints, only one province designated by USAID/Zambia as “rural” could be visited. The team was requested to sample six POs from each of five cohorts. Despite many attempts, a number of scheduling conflicts and declines for interview resulted in the team sampling three POs from Cohort 3. Reasons for declined interviews included POs’ reported lack of allocable time and relevant PO staff being unavailable or no longer affiliated with the organization. To compensate for this, organizations from other cohorts were purposively oversampled. Site visits were ultimately conducted at 35 organizations – five more than originally targeted. In the absence of a random sample selection, the generalizability of findings from qualitative interviews is limited in scope. Additionally, qualitative interviews are inherently subject to specific biases: recall bias, wherein responses are affected by respondents’ ability to recall past experiences; and the Hawthorne effect, whereby respondents aware of the study modify their behavior. The evaluation team used best practices in evaluation to minimize bias and subjectivity to enhance the rigor of the evaluation results. Due to this evaluation’s reliance on qualitative inputs where quantitative data is unavailable, the generalizability of conclusions drawn from qualitative evidence is nonetheless limited in scope. Discussion of service delivery quality, in particular, is based primarily on unverifiable reports of PO staff; thus, significant caveats emerge from the conclusions drawn using this qualitative data. Such caveats are explicitly noted throughout the report where relevant. Due to the inability to access a complete dataset on annual PEPFAR results disaggregated by partner organization, the evaluation team was unable to draw firm conclusions on the link between organization development and HIV/AIDS service delivery within the context of LPCB. However, regression analyses were performed on available data, and where applicable, limitations to the generalizability of the conclusions of such analyses are explicitly stated throughout the report. This limitation was also mitigated 17 through the triangulation of available quantitative measures with qualitative data. Verifiable data on individual PO financial status, such as previous annual funding levels and number of funding sources, was similarly unavailable. This is mainly due to the fact that POs were unlikely to have maintained sound financial records during the years prior to engagement with LPCB. Therefore, analysis of PO financial viability was performed on data gleaned from qualitative interviews. This data, and the analysis thereof, is subject to significant recall bias. Generally speaking, for quantitative data that was available, the evaluation team performed regression and chi square analyses to determine potential correlations and significance levels; however, due to the ex post nature of the evaluation and the lack of valid baseline data or control group, beneficiary impacts were not able to be attributed directly to project inputs. Perhaps the most significant limitation faced by this evaluation – or any evaluation of a capacity building initiative – is the lack of a standardized definition of capacity and its inherent inability to be quantified. For this reason, the evaluation relied heavily on reports from LPCB beneficiaries and stakeholders. Recommendations for improved methods of quantifying specific aspects of organizational function related to capacity are included throughout this report. 18 FINDINGS & CONCLUSIONS 1: ACHIEVEMENT OF LPCB OBJECTIVES KQ 1: To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? FINDINGS Findings for this evaluation question are organized by each project objective and its corresponding activities and results against targets, followed by the project’s overall goal. Objective 1 – Assess: Assess Zambian NGOs, FBOs, and CBOs on their organizational, financial and technical capacity with regards to managing, implementing, results reporting and scale-up of HIV/AIDS programs. Upon enrolment into LPCB, the main project activity under the first project objective – and indeed, the first activity undertaken with partner organizations participating in LPCB – was the administration of the institutional development framework (IDF) self-assessment of capacity. The IDF tool allowed organizations to rate their perceived capacity in different areas of organizational function, such as management resources and financial resource mobilization. Facilitated with the support of a Capacity Leader organization (CL) or Organization Development Facilitator (ODF), the assessment using the Institutional Development Framework was repeated after a year’s engagement with LPCB in an attempt to document organizational growth progress and reinforces the significance of organizational introspection. Under the objective of partner organization assessment, the project tracked the following indicator: INDICATOR TARGET ACTUAL Number of Partner Organizations and Capacity Leaders with increased annual IDF score during the reporting period 80% of 100 target POs 74% (67 of 90 POs reporting at least two IDF scores) Follow-up IDF scores were unavailable for 17 participating POs by the end of the project, making it difficult to conclude whether the target results were indeed achieved. An assumption underlying this indicator is that an annual increase in IDF score is indicative of organizational growth. However, given the inherent subjectivity of an organization’s self-assessment, a change in IDF score cannot be considered an objectively verifiable measure of strengthened capacity. To this extent, the fact that the project fell short of its stated life-of-project goal does not reveal a significant project shortcoming; rather, fewer positive IDF score differences may actually indicate increased PO awareness and understanding of existing capacity gaps. Interviews with partner organization staff confirm this notion, as 31% of POs reported rating themselves “more realistically” during the second administration of the IDF. Of the 35 partner organizations interviewed, 86% POs across all cohorts and provinces reported that the IDF tool was effective in gauging organizational capacity and identifying areas for improvement. The assessment of POs for the purpose of determining and addressing identified capacity gaps may be considered successful to the extent that 80% POs report adapting the IDF for their own specific contexts and continue to implement the IDF regularly after their engagement with LPCB has ended. High uptake and unique adaptation of the tool speaks to the tool’s versatility and value. One Cohort 5 PO reported using 19 the IDF to assess other organizations in its immediate community identified as having similar capacity gaps. Forty-nine percent of POs reported that the distinguishing feature of the IDF is its immediate buy-in from organizations. “It’s a democratic process,” cited staff of one sampled partner organization. One PO reported that it is routine for other capacity building projects to conduct an external assessment of an organization before attempting to build capacity in accordance with externally established benchmarks of success. This PO remarked that the IDF was designed to allow an organization greater control over the direction and quality of its growth than with other capacity building projects, which was cited as a driving factor of its ultimate organizational growth. Reported benefits accrued to POs through LPCB are discussed further in Key Question 3. The IDF’s focus is mainly an examination of the quality of existing institutional components, such as financial reporting and M&E systems. To that effect, assessment of technical capacity to scale-up HIV/AIDS services is not readily assessed using strictly organizational indicators. Level of technical capacity may be deduced from an organization’s self-rating of “service delivery”, “beneficiary participation”, and “monitoring and evaluation”; however, these are singular components falling under and relating directly to the IDF’s overarching “Management Resources” category. Objectives 2 and 3 – Strengthen technical and institutional capacity: Strengthen the operational, management and technical capacity of local NGOs, CBOs, FBOs, and networks so that they can expand and improve the quality of their HIV/AIDS services and activities, link to service delivery networks, and advocate effectively for legislative and policy change. Support local partners and sub-partners with the technical assistance and financial resources to put effective management and financial systems in place, develop and implement business plans, and manage organizational change. The first project activity undertaken to strengthen technical and institutional capacity was to train individuals from partner organizations in a series of core organizational functions aimed at building critical skills necessary for an organization’s operation. Under this activity, LCBP tracked the following indicator: INDICATOR TARGET ACTUAL Number of Health Care Workers who Successfully Complete an In-service Training Program within the Reporting Period 1,380 Individuals 8,937 Individuals (4,116 Males; 3,898 Females) This PEPFAR indicator combined the number of individuals trained through LPCB core and elective trainings, as well as the number trained in HIV/AIDS technical areas through LPCB service delivery grants. Through core and elective trainings only, LPCB reported training a total of 1,915 individuals over the life of the project. The remaining 7,022 individuals trained through service delivery grants were reported to have been trained by partner organizations implementing service delivery grants with activities related to training community beneficiaries (e.g. peer education). With respect to the number of health care workers trained, both directly by LPCB and by partner organizations, the project exceeded its target; however, it should be noted that “health care worker” in the context of LPCB was defined as a non-clinical worker involved in service delivery, program support and management, or OD, as opposed to health care workers in the clinical PEPFAR sense. Further, individuals were counted more than once if they attended more than one training, indicating a serious data quality issue. Because it is not immediately discernible how many individuals were trained, it is not possible to verify the extent to which these indicators met their targets. Individuals reported to have been trained through partner organizations directly were expected to be trained using materials provided through LPCB; however, the extent to which the quality of such training remained consistent with what was provided 20 through LPCB is not similarly unverifiable. Results from this activity reveal the effectiveness of utilizing LPCB trainees as trainers of community beneficiaries. LPCB provided direct training to less than a third of the total number of individuals who were ultimately reported to have been trained in HIV/AIDS technical competencies, indicating a significant multiplier effect of LPCB’s training inputs. One Cohort 1 PO remarked, “Now we can multiply our results through peer educators.” A critical aspect of this activity – and, indeed, LPCB generally – was to provide training in HIV/AIDS service delivery competencies and technical skills aimed at meeting the project’s ultimate goal of increased HIV/AIDS quantity and quality of services. Over the life of the project, LPCB offered partner organizations nine core trainings (the ninth training, “HIV/AIDS Technical Update”, was introduced for Cohorts 3, 4, and 5) and a “tasting menu” of 12 elective trainings. Prior to the addition of the HIV/AIDS Technical Update workshop, Social and Behaviour Change Communication was the only core training offered in a technical competency. Overall, partner organizations sampled for interview rated the quality of LPCB workshops highly; 63% of POs cited trainings as LPCB’s most valuable component. Of the 35 POs interviewed, 7 reported that LPCB’s workshops were, in fact, higher quality than similar trainings as part of other projects. In terms of usefulness, the most highly valued trainings were reported to be Resource Development and Monitoring and Evaluation. Of note, POs sampled from early cohorts rated the Social and Behavior Change Communication training as particularly useful to building capacity, whereas those sampled from later cohorts were more likely to rate the HIV/AIDS Technical Update training as particularly useful. The findings indicate that, while not the principle focus of LPCB workshop inputs, demand was high among all POs for technical competencies related to HIV/AIDS service delivery. Further, one of the most highly desired follow-ons to LPCB was refresher training in technical skills; 89% POs reported the need for continued capacity development in technical areas. High residual demand for technical skills suggests that the amount of training offered by LPCB in these areas was insufficient to meet demand. Alternatively, POs may now have the capacity to recognize the value and utility of technical training in response to increased institutional capacity, and are able to demand additional training given that awareness of capacity gaps is understood to be greater in the wake of LPCB than before. Interviews with sampled POs revealed that organizations of lower starting capacity reported being disadvantaged by LPCB’s “one size fits all” training approach, whereby organizations with varying levels of capacity were trained together. Twenty-three percent of POs reported that training material was often too advanced for their level of understanding, suggesting that some information may not have been absorbed as effectively for small POs than larger ones. This may have impacted the results of community beneficiaries trained by POs; 3 POs reported not being able to fully apply the skills they had learned from LPCB as a result of the advanced level of material presented to them. A second key activity aimed at bolstering capacity development was the promotion of collaboration and networking among POs. Increased opportunities for partnerships, in theory, promotes strengthened organization through knowledge exchange, streamlined activity implementation, and funding partnerships. Networking opportunities offered through LPCB included Cohort launch workshops, Provincial PO Cluster Meetings, PO exchange visits, and an All-partners’ Conference. Under this activity, LPCB reported on the following indicator: 21 INDICATOR TARGET ACTUAL Number of networking opportunities offered to stakeholders during the reporting period 30 POs provided networking opportunities 62 POs provided networking opportunities Over the life of the project, LPCB exceeded the targeted number of POs offered networking opportunities by 206%. Thirty-one percent of POs sampled reported that networking was one of the most valuable components of LPCB, and thirty-five percent of POs citing the significance of networking to the sustainability of their organization’s growth. Contributing to the success of this particular activity was the eagerness of organizations – particularly rural – to learn from the experiences of other, more successful organizations. The extent to which POs valued networking opportunities is evidenced by the number of spin-off collaborations built by between POs outside of LPCB; 14 POs cited LPCB’s encouragement of networking as a catalyst for securing additional partnerships with organizations unaffiliated with LPCB. Implications of increased PO networking are discussed further under Key Question 5. Objective 4 – Fund: Grants provided to POs to strengthen organizations and expand HIV/AIDS services POs were provided funding opportunities through commodity grants and service delivery grants. Funding was intended to encourage organizations to apply skills gained from training to account for funds received, while also implementing and measuring the impact of proposed HIV/AIDS activities. Under this activity, the project tracked the following indicators: INDICATOR TARGET ACTUAL Number of POs that received grants through LPCB during the reporting period 50% Partner Organizations and Capacity Leaders 90% Partner Organizations and Capacity Leaders Number of PO grant recipients that achieve at least 75% of their stated objectives/targets during the reporting period 27 POs 43 POs (of 56 POs) The project exceeded its target in terms of number of grants provided to beneficiaries: 52 received commodity grants, 56 received service delivery grants, and 12 received both. As reported by LPCB staff, a factor driving the success of this indicator was the unexpected volume of organizations identifying the need for basic commodities. Though baseline target never set, project staff also reported that the value of service delivery grants offered was lower than originally anticipated due to lower PO financial absorption capacity than expected. The number of service delivery grant recipients achieving at least 75% of their stated objectives was also exceeded over the life of the project, suggesting additional project success by way of strengthened technical capacity among POs. The volume of grants provided by LPCB speaks to the effectiveness of the training received by LPCB beneficiaries to an extent, as 54% of POs reported being able to apply for service delivery grants as a direct result of increased skills gained from training. The quality of LPCB’s grant component was also cited to be highly valued among POs. All POs sampled who were recipients of commodity grants reported that commodities received were highly impactful on the operation of the organization. Recipients of service delivery grants among POs sampled for interview also cited the utility of the grant in helping to achieve organization objectives. Fifty-four percent of POs also noted the value of the process of applying to grants in the first place, as POs were able to harness guidance from their ODF/CL to compose a quality proposal. Despite the quantity of grants provided and reported benefits accrued therefrom (discussed further under Key Question 3), challenges reported by POs related to the quality of grant management negatively impacted project results. Firstly, 91% of POs sampled who were recipients of either type of LPCB grant reported that the most significant challenge faced with LPCB as a whole was the delayed disbursement of 22 funds. In each case, the PO cited a delay of up to three months, causing a subsequent delay in implementation of proposed activities. Thirty-one percent of these POs also reported LPCB’s expectation that activities be implemented in accordance with the originally proposed timeline, causing POs significant financial stress and delayed activity implementation. Additionally, 34% of POs remarked that the prescribed period of grant performance was insufficient to effect a significant impact on the community. Project staff from a Cohort 2 PO stated they “wished [they] could have done more” with the funding received, reiterating the bearing delayed disbursement had on implementation. According to LPCB project staff, challenges with funding disbursement stemmed from the sequencing of the financial report review process. Funds were reportedly unable to be released until a PO’s first monthly report passed through a series of internal quality checks, often amounting to 30 days or more. An additional challenge impeding the achievement of project results was found to be the allocation of grant funds for activities only. Eleven POs reported feeling constrained in their ability to motivate staff members or volunteers with financial incentives; funding provided from LPCB was tied directly to activities; thus, POs implementing additional activities with service delivery grants cited struggles with retaining staff and volunteer engagement. Considering the role that PO volunteers peer educators and caregivers play in the multiplication of service delivery results throughout the community, poorly motivated volunteers represent a significant threat to the sustainability of project results. Objective 5 – Document: Assist local organizations to document impact and disseminate evidenced-based innovations, best practices and lessons learned. A key project component of strengthening technical capacity was an emphasis on monitoring and evaluation as a means for an organization to document its activities and remain accountable to beneficiaries and donors. Training in M&E and reporting on service delivery grant implementation were the main LPCB activities undertaken to strengthen PO ability to document community impact. Under this objective, LPCB tracked the following indicator: INDICATOR TARGET ACTUAL Number of POs and CLs that submitted data to the National AIDS Council (NAC) during the reporting period 75% POs and CLs 88% POs and CLs Submission of data to the National AIDS Council (NAC), though not required for LPCB beneficiaries, was explicitly encouraged through the M&E core training and guidance of CL/ODFs. The project exceeded its target for the number of beneficiary organizations reporting the submission of data to NAC; however, many POs reported inconsistent reporting over the life of the project. Contributing to the success of this project result was the quality of the M&E training; as aforementioned, 63% POs cited this training as particularly beneficial to building skill sets and imparting the significance of data tracking. Of the 35 POs sampled, 17% reported the absence of an M&E system prior to LPCB. These POs cited the establishment of functional M&E systems as a direct result of LPCB’s M&E training. The effectiveness of M&E training in imparting the significance of impact documentation is evidenced by POs’ contribution to reported results under USAID/Zambia’s HIV/AIDS Multisectoral Results Framework – specifically, the strategic objective of reduced HIV/AIDS impact through a multisectoral response – as well as Zambia’s annual PEPFAR targets. POs sampled for interview reported using data for the following purposes: Program evaluation and resource mobilization Improve programming through strategic expansion of activities Demonstrate impact to donors 23 Report to District AIDS Task Force (DATF) As expected, POs with lower relative capacity noted challenges with maintaining timeliness and consistency of reporting. These challenges included conforming to a high level of report detail and poor access to internet. One notably reported challenge was the requirement of LPCB sub-grantees to report on PEPFAR indicators that were not always aligned with a PO’s activities during the reporting period. For example, one Cohort 3 PO reported on specific indicators related to its children’s programming, which were not able to be captured by the requisite PEPFAR indicators comprising monthly LPCB reports. This resulted in the appearance that POs had a reduced community impact than actually achieved. Objective 6 – Sustain: Create in-country capacity to provide technical support for organizational strengthening. The final strategic objective of LPCB involved training existing local organizations and individuals to provide on-going capacity building support to partner organizations. LPCB tracked the following indicators for this objective: INDICATOR TARGET ACTUAL Cumulative number of Capacity Leader Organizations (CLs) participating in LPCB 6 CLs 4 CLs Number of health care workers (from CLs) who successfully completed an in-service training program within the reporting period 70 ODFs 152 ODFs Number of independent health care workers (Organization Development Facilitators (ODFs)) who successfully completed an in-service training program within the reporting period 34 ODFs 13 ODFs Number of ODFs licensed to facilitate the IDF assessment with HIV/AIDS matrix 15 ODFs 13 ODFs Number of local organizations to which CLs/ODFs provide training or technical assistance 100 organizations 107 organizations Establishment of a professional organization development association in Zambia Established by FY 2011 Established in 2010 Results for indicators under this objective are mixed. Targets were exceeded for the number of organizations receiving technical assistance, as well as the number of individuals within Capacity Leader Organizations trained to provide technical assistance. However, the number of Capacity Leader Organizations enrolled in LPCB and the number of individuals trained as independent Organization Development Facilitators (ODFs) fell below LOP targets. Again, a data quality issue arises due to counting ODFs twice if they attended more than one training. Because it is not immediately verifiable how many individuals were trained in absolute terms, it is not possible to conclude whether these indicators met their targets. Failure to meet the benchmark of recruited Capacity Leader Organizations, however, was reported to have been due to a conscious decision on the part of project staff to limit their number. Quality, according to the recipients of their technical assistance, was reportedly very high: 79% of POs sampled reported that the quality of technical assistance received either met or exceeded expectations. One of the factors cited to have influenced the perception of CL/ODF quality was the consistency of follow-up and availability of support; POs noted that the explicit mentorship of their designated CL/ODF, as opposed to informal, infrequent “check-ins”, raised the quality of technical assistance beyond what had been received previously from other projects or organizations. Providing further evidence of the quality of technical assistance were reports from POs that their CL/ODF remained committed to OD process 24 facilitation rather than “hand-holding”. Several POs noted that their own expectations of ODF support evolved as a result of ODF insistence on PO ownership of the OD process. These POs, whose initial dissatisfaction with their ODF’s assistance evolved into appreciation, indicate to some extent LPCB’s success in changing perceptions of the value of OD among beneficiaries. Comparative advantages of CLs and ODFs are further discussed in Key Question 2. In addition to training local individuals and institutions to provide technical assistance for LPCB specifically, LPCB sought to reinforce the sustainability of country ownership through the establishment of the Institute of Organization Development Facilitators (IODF) in 2010. Since its inception, membership of the IODF has grown to include non-LPCB actors and organizations with a stake in capacity building in a range of sectors beyond HIV/AIDS. To the extent that the IODF represents a nationally-recognized, stand-alone organization promoting the principles of organization development replete with its own stated mission and objectives, a key component of LPCB’s sustainability objective has been met. The establishment of the IODF has similarly helped achieve one of LPCB’s implicit goals of raising the visibility of organization development among public and private sector stakeholders through stakeholder conferences and publications of standards and best practices. However, the utility of the IODF as an OD service provider for capacity-challenged organizations is limited in an environment in which funding explicitly for institutional development remains difficult to attain. The sustainability of the IODF is addressed further in Key Question 5. LPCB Goal – Increased and better-quality HIV/AIDS service delivery by LPCB Partner Organizations in Zambia To measure results against LPCB’s overarching goal of increased quantity and quality HIV/AIDS service delivery, partner organizations reported on PEPFAR indicators specific to prevention and health systems strengthening. The main prevention indicator tracked over the life of the project, P8.1D, measured the number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. Annual results against targets for this, as well as subset indicators, are depicted below. Figure 1. PEPFAR Indicator Results Reported by LPCB Beneficiary Organizations* 25 As evidenced above, annual performance against project indicators declined over the life of the project, and annual targets for the main prevention indicator and its subset “Prevention – Abstinence/Being Faithful (AB)” were not met in LPCB’s final year. Declining numbers of people reached with preventive HIV/AIDS services, however, does not necessarily indicate poor project performance. Instead, decreasing performance for prevention indicators may be due in part to the application of evidence-based strategies on the part of partner organizations, which specifically encouraged a greater emphasis on more focused interventions for wider target population groups; concurrent with LPCB implementation, PEPFAR’s emphasis on abstinence/being faithful messaging began to shift to a focus on a wider range of preventive services and populations (such as people living with HIV (PLHIV)). Similarly, over the life of LPCB, PEPFAR began to strongly emphasize the delivery of preventive services through individual or small group-focused interventions over the type of mass prevention campaigns POs previously utilized. The cumulative impact of such high-level policy shifts was absorbed into the training and technical assistance provided to LPCB beneficiaries, which may have affected PO service delivery. Indeed, qualitative data gleaned from sampled POs confirms that LPCB training and assistance influenced the manner in which POs delivered services – specifically, by conforming to evidence-based strategies for community HIV prevention activities. The main driver of declining results over time, however, is more likely to be the nature of cohort composition and term of PO engagement in the project. Due to the design of LPCB cohorts – each cohort’s initiation staggered by a year with a term of engagement of one to two years – those POs reporting on PEPFAR indicators during Year 1 and 2 are different organizations than those reporting during Year 4 and 5. By their very nature, POs enrolled in LPCB varied widely in terms of individual target populations, types of services provided, geographic locations, sizes, and existing capacity levels. For this reason, coupled with each PO’s engagement for a portion of the total project duration, results from early and late project years are not directly comparable. Further contributing to a decline in results reporting over the life of the project is the fact that later cohorts of partner organizations were comprised of higher proportions of smaller, rural organizations than earlier cohorts. Situated in less populated areas, these POs are naturally constrained by the number of people to whom preventive services may be offered, resulting in lower aggregate results reporting in later years of LPCB. Figure 2. Gender Disaggregated PEPFAR Indicator Results Reported by LPCB Beneficiary 0 5000 10000 15000 20000 25000 30000 35000 40000 M F M F M F M F 2010 2011 2012 2013 No. People Gender Disaggregated PEPFAR Results PwP Prevention Prevention AB Prevention MARP Prevention Gender HCW Trained 26 Reported LOP results appear to remain consistent among men and women across all PEPFAR indicators. However, this is not to say that POs were equally successful at targeting both men and women. Instead, the main drivers of prevention services for women were women-led, women-focused organizations whose express missions concerned female-targeted service provision. Figure 3. LOP Aggregate PEPFAR Results against Targets Overall, LPCB exceeded its LOP targets for the main prevention indicator, its subset indicators, and the number of health care workers trained. Partner organizations were most successful in their delivery of Prevention AB, followed by Prevention with Positives (PwP) and Prevention among MARPs. As previously mentioned, some POs reported difficulty in fitting their activities into PEPFAR indicators, particularly those POs providing services to vulnerable populations apart from those explicitly defined as MARPs. Hence, reported results do not necessarily reflect the full range of HIV services provided by POs to beneficiary communities. The effects of organization development on PO service delivery are explored further in Key Question 4. CONCLUSIONS With the exception of two unverifiable measures, the project achieved its target indicators and planned activities. Overarching project objectives of organizational assessment, strengthened institutional capacity, funding, and documenting impact were largely met by the project activities. The IDF is largely focused on capturing measures of institutional capacity; thus, it does not readily allow assessment of an organization’s technical capacity. POs’ short length of engagement with LPCB makes it difficult to determine the success of the project’s sustainability objective. The appearance of reduced success against targets over the life of the project may be due in small part to the effectiveness of LPCB technical training strategies, but is largely a reflection of cohort composition and individual partner organization focus. 27 LESSONS LEARNED Future capacity building projects should incorporate the use of external measures of organization growth into eligibility criteria used to enroll POs. The absence of objective data makes it difficult to assess actual organizational growth over time. The effects of strengthened capacity may spill over into areas of organizational function not easily captured using PEPFAR metrics. Thus, low PEPFAR reporting over the life of the project is not necessarily indicative of low performance or low capacity. 28 2: LPCB DESIGN, IMPLEMENTATION, AND MANAGEMENT KQ 2: To what extent were the project design, implementation, and management effective and why? FINDINGS What are the best practices and lessons learned for each project phase? PO Enrolment Best Practice: Conduct verification site visits of POs applying for capacity building assistance to mitigate the potential for organizations to misrepresent their capabilities or intentions. Lesson Learned: During the enrolment phase of LPCB, POs submitting expressions of interest were assessed by project staff to determine their eligibility to participate in the project. The first two cohorts of LPCB beneficiaries were assessed using written applications only. Project staff soon learned that this practice led to the enrolment of so-called “briefcase organizations”, whose applications described significant aberrations from reality. Such organizations, deemed inappropriate for LPCB assistance, were filtered out of future cohorts when the project began employing site verification visits for promising LPCB candidates seeking enrolment. A key lesson learned during this phase was the propensity for organizations to exaggerate their existing capacity level in expectation of receiving funding. Lesson Learned: LPCB was designed to engage POs in rounds of five overlapping cohorts comprised of roughly 20 POs each. While this practice allowed for the engagement of a large number of organizations over the span of five years, the management of distinct cohorts at different phases of project implementation caused significant administrative strain. In response, LPCB added additional staff commensurate with the number of POs enrolled to attempt to address PO inquiries in a timely fashion. Lesson Learned: LPCB Cohorts were purposely designed to act as communities of practice, comprised of organizations of varying sizes, locations, and levels of capacity. While envisioned to promote valuable knowledge exchange between these organizations, POs reported this design was not conducive to their own capacity growth. Smaller POs, in particular, reported feeling disadvantaged by being grouped with organizations capable of advancing through project phases faster than them. Training Best Practice: The ability of POs to follow-up directly with workshop facilitators after trainings contributed to their overall effectiveness. Trainings were found to have been additionally effective due to the interactive, participatory nature of workshops. Lesson Learned: LPCB core and elective trainings were conducted for POs by cohort, such that organizations of varying capacity levels within cohorts received the same instruction and materials. POs sampled for interview reported challenges with this arrangement, citing that the information in trainings was too advanced for organizations of lower capacity. Indeed, many smaller POs noted that the “one size fits all” approach to training inhibited their ability to absorb, and ultimately apply, core competencies in institutional and technical capacity. 29 Grants Best Practice: As opposed to other capacity building projects focused on providing rapid funding for organizations, LPCB engaged POs for approximately 10 months before grant opportunities were even offered. This allowed the project to reinforce its focus on building strong institutions and ensuring POs’ priorities were aligned with the project. Best Practice: LPCB’s grant component can be considered a best practice to the extent that it helped to strengthen the financial capacity of organizations. LPCB provided POs with proposal feedback in a constructive atmosphere aimed at improving POs’ financial viability. The practice of applying for funding through LPCB was cited as one of the most valuable components of LPCB among sampled POs. Best Practice: Innovative grants, which were offered to six POs determined to be among the highest-performing over the life of the project, were reported by sampled POs to have been highly impactful on the community. The grant’s explicit goal of increasing the quality and delivery of HIV/AIDS services to PO beneficiaries encouraged recipient POs to propose “innovations”, such as “Youth Friendly Corners”, and the establishment of clinic referral systems, which were reported to have improved the quality of services provided to the community. In the absence of an LPCB quality assurance mechanism to verify the delivery of quality HIV services, it is not possible to determine that the quality of service delivery was indeed affected by LPCB inputs. However, the explicit encouragement of innovation in service delivery was found to have influenced POs to apply newfound organizational and technical skills toward benefitting community stakeholders. Lesson Learned: Service delivery grants ranged from one to two years in duration, depending on a POs’ ability to successfully acquire a grant extension. Of the POs sampled for interview, 61% of recipients of service delivery grants reported that the grant’s period of performance was not long enough to produce meaningful results in the community. By the end of the project, it is difficult to ascertain the long-term sustainability of capacity gains made post-LPCB as a result of the reportedly short length of PO engagement. Lesson Learned: Ninety one percent of sampled POs who received a grant through LPCB sampled reported delays in funding disbursement that negatively impacted the delivery of services. The primary bottleneck driving disbursement delays was found to be a lengthy financial report review process at the project level that resulted in delays of up to three months in some cases. Phase-out Lesson Learned: Thirty-seven percent of sampled POs reported confusion surrounding the role of their assigned CL or ODF post-LPCB. Lack of clarity regarding the availability of technical support post-LPCB was reported to have caused strain on POs requesting additional mentorship from ODFs no longer funded to provide technical assistance. What changes occurred during LPCB implementation that may have had a bearing on activity outputs and outcomes? During its implementation, LPCB adopted a series of additions and modifications to activities in response to recommendations identified in LPCB’s mid-term assessment. First, LPCB moved to accelerate the implementation of all planned activities and complete its Performance Monitoring and Evaluation Plan (PMEP), which was still incomplete as of 2010. LPCB consequently accomplished all planned activities by the end of the project; however, the speed with which implementation occurred may have negatively impacted the quality of outcomes, as Cohort 5’s shortened length of engagement impeded their ability to meet 30 established graduation criteria. POs sampled from Cohorts 4 and 5 reported dissatisfaction with the amount of time they were allotted to implement grants and report achievements. A notable addition to LPCB was the implementation of graduation plans for beneficiaries, which established benchmarks for success. The establishment and implementation of graduation plans contributed to overall project results by incentivizing success (graduated POs were recommended to external donors) and providing an additional measure by which to categorize an organization’s growth as a result of LPCB. A number of changes taking place external to LPCB during the project’s implementation had a bearing on activity outcomes as well. Most notably, the suspension of Academy for Educational Development (AED) in the fall of 2010 and the subsequent management transition to FHI360 created delays in activity implementation and funding for recipients of LPCB grants. FHI360 was awarded a no-cost extension to compensate for these delays, which prolonged implementation by six months. A significant external factor potentially impacting the achievement of project results is the phenomenon of donor fatigue. Funding commitments toward HIV/AIDS initiatives globally have slowed during the past decade due to a reduction in perceived urgency surrounding the epidemic, which has been compounded by the recent economic downturn.1 This notion was independently confirmed by POs reporting that the perceived level of commitment from funders, as well as the overall number of funders in the HIV/AIDS arena has declined in recent years. Though financial trend data is unavailable to verify that PO funding levels have been affected by external factors, POs reported the perception of fewer financial resources currently available than in years past. Donor fatigue has significant implications on the achievement of LPCB outcomes related to financial viability; the inability of some POs to achieve greater donor diversification may be more a function of the current donor atmosphere than low PO capacity. An emergent shift in PEPFAR focus during the years of LPCB implementation resulted in renewed emphasis on small group and individual-level HIV/AIDS interventions over mass campaigns, as well as a move away from abstinence/being faithful messaging to interventions targeted toward more inclusive vulnerable populations (e.g. PLHIV). The aggregate result of these high-level changes may be reflected to some extent in the gradual decline in LPCB’s overall annual results against PEPFAR indicators. On the surface, shifted PEPFAR focus appears to negatively impact LPCB results; however, declining PEPFAR results over time may in fact indicate the degree to which LPCB effectively encouraged the application of evidence-based strategies among POs. By focusing on fewer people within more specific target populations, some POs reported having raised the quality of HIV/AIDS service delivery in their communities. It should be noted, however, that quality of service delivery is, again, impossible to assess objectively in the absence of a quality assurance system verifying PO activities. The extent to which service delivery was affected by LPCB is further explored in Key Question 4. Over the life of the project, LPCB did not operate in a vacuum; rather, LPCB was one of many Zambian initiatives providing capacity building support of some kind to beneficiary organizations. The existence of concurrent support from other projects may have aided in effectively accelerating the achievement of LPCB results. Because the concept of capacity building support was not new for a majority of POs, the specific inputs of LPCB may have been able to gain traction among beneficiaries more easily than if concurrent support for capacity building were not already a component of the Zambian donor environment. 1 Grepin 2011. “Efficiency considerations of donor fatigue, universal access to ARTs and health systems” 31 How effective were tools used to track organizational capacity and were they good predictors of organizational success? The main tool employed by LPCB to track changes in organizational capacity was the Institutional Development Framework. The IDF, as previously described, is a self-assessment used by organizations to rate perceived capacity in a number of dimensions such as management, M&E, and financial viability. The IDF was found to be one of the most highly valued components of LPCB; 86% of POs sampled reported that facilitation of the IDF provided an accurate representation of their organization’s capacity. A further indication of the IDF’s value among POs is the fact that 80% of POs report having adapted the IDF for their own use with the intention of assessing capacity on a regular basis. While the IDF is an effective tool in a qualitative sense, its resulting score cannot be considered a valid, quantifiable measure of an organization’s capacity. Self-assessments are inherently subjective, as organizations may want to exaggerate certain measures to appear more capable of absorbing funding. On the other hand, an organization whose capacity has been strengthened may rate itself more honestly in subsequent assessments, reducing the appearance of capacity gains. For these reasons, the IDF alone is an unreliable tool for quantifying organizational development. In exploring whether the IDF score is a good determinant of success, the evaluation team found no statistical correlation between change in capacity score and PO graduation status. LPCB’s post-graduation assessment survey, conducted on a sample of POs six months after graduating, provides an opportunity for capturing a more quantifiable representation of an organization’s change in capacity. The survey was used to gain information on specific indicators relevant to an organization’s operation, such as the frequency of Board meetings and the number of funding sources. This type of tool differs from the IDF in two significant ways. First, the specificity of indicators allows for a direct comparison in values over time, as opposed to an aggregate score comprised of a number of qualitative measures. Secondly, greater objectivity is preserved in being administrated by an individual external to the organization. With the addition of outcome indicators related to other organizational phenomena, such as staff turnover and frequency of strategic planning, the survey may be used as an external baseline and end line capacity assessment. While the IDF remains a valuable and effective tool, a survey modeled after the post-graduation assessment survey may provide a more accurate measure of an organization’s capacity level. How effective and efficient were Capacity Leader Organizations and Organization Development Facilitators, and what are their comparative advantages? Overall, the quality of technical assistance provided by Capacity Leaders and individual Organization Development Facilitators was rated highly; 79% of POs reported that expectations were either met or exceeded by their CL/ODF. While there is no discernible difference in outcomes between recipients of either CL or ODF support, POs reported a number of advantages and disadvantages of the two types of TA. It is important to note that POs only received TA from one type of facilitator over the course of its engagement with LPCB; thus, POs themselves were unable to report direct comparisons between CLs and ODFs. Of the POs sampled for interview, 57% and 43% of POs received assistance from a CL and individual ODF, respectively. The most frequently cited strengths of CLs were: comprehensive, high quality technical assistance, the high responsiveness to the individualized needs of POs, and ability to access a pool of resources from within the CL organization. Frequently cited weaknesses of CLs tended to relate to poor timeliness of feedback and inconsistent communication; however, the majority of weaknesses reported were in reference to particular CLs, not all. POs with prior relationships with their assigned CL were more likely to be critical of the 32 technical assistance received. For instance, two such POs reported feeling “overlooked” when asking for specific assistance, and attributed poor responsiveness to the CL’s assumed knowledge the PO’s actual need for TA. Individual ODFs were seen as particularly strong in the specialization of their expertise, responsiveness to PO needs, and availability of support when needed. Many POs noted that their ODF was “just a phone call away.” A critical disadvantage cited frequently was the lack of ODF technical expertise in specialized areas related to HIV/AIDS service delivery, such as maternal and child health. Despite the fact that ODFs were defined primarily by the project as organization development experts – not technical specialists – POs noted that increased technical support from their ODF in HIV/AIDS competencies would have improved their overall utility. Further, many POs remarked that ODFs were inherently challenged in their ability to house expertise in every area; two POs reported, “One person can’t be an expert in everything.” Other ODFs were criticized for not “thinking outside the box.” Again, larger POs were more likely to list criticisms of their assigned ODF, presumably due to higher existing capacity level and resulting higher expectations of technical assistance. Interviews with smaller POs, on the other hand, tended to reveal overwhelmingly positive reviews of technical assistance – both from CLs and ODFs. Smaller organizations, less likely to have previously received similar types of technical support from other projects, had very little to compare the quality of technical assistance to, and thus, reported higher levels of satisfaction than larger organizations. The basis of major criticisms of ODFs was confirmed directly by one ODF, self-described as “businessman at heart.” This ODF acknowledged shortcomings in technical areas of relevance to HIV/AIDS service delivery, but maintained that institutional capacity from a business perspective can still be of value to POs with capacity deficits, particularly with the assistance of the newly-established Institute of Organization Development Facilitators (IODF). The reason ODFs could provide assistance under LPCB at all, he explained, was that the project was primarily about organization development, not service delivery. This point underscores the existence of a disconnect in project logic between organization development and service delivery, calling into question the utility of HIV-specific technical expertise for a project perceived to be purely process-oriented. The evaluation team’s interview ZHECT provided valuable insight into the comparative advantages of CLs and ODFs, as ZHECT had the unique experience of being both an LPCB partner organization and Capacity Leader. ZHECT staff reported that TA from a Capacity Leader is naturally more beneficial to POs since an organization contains a full cadre of resources to pull resources from, as opposed to an individual ODF with limited knowledge. A CL also serves as an organizational mentor, reported ZHECT, and POs benefit from having an organization to “look up to.” One Cohort 2 PO lent credibility to ZHECT’s position, stating: “We discovered that other organizations were able to surpass us because they were assigned to a CL instead of an ODF, like us.” Findings from sampled POs reveal higher overall levels of satisfaction for CLs than ODFs. CONCLUSIONS Smaller POs were disadvantaged by the design of cohort groupings. LPCB demonstrated responsiveness to lessons learned over the life of the project and effectively addressed project gaps identified by the mid-term assessment. Overall project effectiveness was negatively impacted by changes in project management (AED to FHI360), funding delays, short PO engagement, and donor fatigue. While the IDF is an effective tool in a qualitative sense, it is a poor determinant of organizational success. The LPCB post-graduation assessment survey modeled may be more effective tool for quantifying organizational growth. 33 Capacity Leaders Organizations were perceived as more effective than individual ODFs. LESSONS LEARNED Knowledge exchange among POs of varying levels of capacity can be mutually beneficial; however, grouping such different organizations together in project implementation rounds projects the notion that POs of lower capacity are expected to perform at the level of higher capacity POs. Homogenous grouping for trainings is more conducive to PO learning and sustained morale. Technical assistance is highly valued by POs, regardless of whether it is packaged as a CL or ODF. 34 3: BENEFICIARY ANALYSIS OF OUTCOMES KQ 3: To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? FINDINGS Linking Inputs with Key Outcomes The extent to which partner organizations experienced measurable changes in capacity as a result of LPCB is best assessed within the context of LPCB strategic objectives and their intended results. The underlying project theory assumes that trainings and technical assistance will lead to increased beneficiary knowledge (sensitization), increased knowledge will be applied toward establishing and conforming to institutional systems and standards (behavior change), and improved systems will lead to greater financial viability through the successful acquisition of LPCB and external grants (increased capacity). To assess measurable changes in the capacity of LPCB beneficiaries, the evaluation team aligned LPCB objectives with key outcome indicators, which were disaggregated and averaged by cohort and province. Tables 1 and 2 below display matrices of project inputs and linked beneficiary outcomes for all 103 POs; figures are averaged for each cohort and province to account for differences in sample size. All data is sourced from the LPCB project database and final project reports provided by MSI and FHI360 project staff. 35 Table 4. Key Project Inputs and Linked Outcomes by Cohort Cohort Avg # Trainings Attended Avg USD Grant Value Avg IDF Score Change % POs with Increased IDF Score % POs Graduated % POs Reporting Non-LPCB Grant Oversight/ Vision Mgmt HR Financial Resources External Resources Total 1 15 47,356 +35% +47% +22% +46% +43% +39% 94% 28% 89% 2 14 41,463 +17% +38% +47% +107% +25% +36% 57% 35% 81% 3* 14 63,519 +4% +10% +8% +9% +4% +6% 69% 37% 80% 4* 14 42,103 +3% +14% +30% +3% +11% +12% 67% 27% 81% 5* 18 12,633 +23% +18% +18% +15% +10% +16% 93% 10% 76% *Unavailable endline IDF scores omitted from average Table 5. Key Project Inputs and Linked Outcomes by Province Province Avg # Trainings Attended Avg USD Grant Value Avg IDF Score Change % POs with Increased IDF Score % POs Graduated % POs Reporting Non-LPCB Grant Oversight / Vision Mgmt HR Financial Resources External Resources Total Central* 15 16,459 +25% +32% +38% +7% +16% +25% 100% 14% 50% Copperbelt* 16 26,889 0% +20% +24% +11% +9% +12% 63% 45% 100% Eastern* 16 32,141 +29% +27% +48% +41% +19% +28% 85% 21% 86% Luapula 13 57,615 +5% +14% +13% +17% +4% +9% 56% 30% 80% Lusaka* 14 90,387 +9% +7% +14% +16% +6% +8% 72% 41% 90% Muchinga* 16.5 9,234 +9% +25% +52% +23% +11% +22% 100% 0% 50% Northern** 15 17,203 -25% -9% +5% 0% -3% -7% 0% 33% 33% Northwester n 15 21,309 +8% +10% +17% +6% +6% +8% 89% 0% 89% Southern* 16 38,469 +12% +17% +57% +33% +10% +19% 80% 38% 62% Western* 17 10,887 14 7 0 14 12 7 80% 14% 100% *Unavailable endline IDF scores omitted from average **Endline IDF scores of only one PO available 36 The above data reveals the following noteworthy points regarding beneficiary outcomes: 1. Due to its inherent subjectivity, change in IDF score over the life of a PO’s engagement with LPCB is not alone a valid, quantifiable measure of a PO’s actual change in capacity. However, it does appear that earlier cohorts rated themselves higher than later cohorts, suggesting that the quality of IDF administration and level of PO understanding of the IDF process may have improved as the project evolved. Human Resources and Management are the two highest rated improvement areas reported by POs across cohort and province, indicating LPCB’s effect on perceived improvements in operational and management capacity. Despite perceived incentives for POs to exaggerate their financial capacity, IDF scores in this area appear to remain consistent with gains in other rating areas. 2. A potential inverse relationship exists between the number of trainings attended by a PO and its graduation status. This could suggest that POs of lower capacity recognize critical capacity gaps and elect to take more trainings than their higher capacity counterparts. Attending more trainings is not necessarily sufficient to raise a PO’s capacity to the level at which they can to successfully attain LPCB grant, however, which explains the corresponding lower graduation rates for cohorts and provinces comprised of lower capacity POs. Additionally, LPCB introduced new elective trainings over the life of the project, such that POs in later cohorts, who were more likely to have lower capacity were simply provided more trainings to choose from. 3. Data reveals a link between grant value and graduation status – cohorts and provinces given higher grant amounts also have higher graduation rates – however, this relationship should not be interpreted as causal. POs who received high value grants were awarded on the basis of having met a specific capacity threshold, meaning those POs were already more likely to graduate. Lower value grants were commodity grants, which were designated for lower capacity POs, explaining why graduation rates among later cohorts and more rural provinces are low. More urbanized provinces received more grant money and had higher graduation rates, but this is presumed to be due to higher levels of starting capacity. 4. Overall, graduation rates are highest among urbanized provinces and earlier cohorts. The graduation rate for Cohort 5 is significantly lower than other cohorts, driven both by lower overall existing capacity, as well as a shortened engagement with LPCB; delays in LPCB implementation resulted in a condensed timeframe for Cohort 5, which may have impacted POs’ ability to meet full graduation criteria by LPCB’s closeout. 5. All cohorts report high rates of grant attainment beyond LPCB. Generally, more urbanized provinces have higher rates of non-LPCB grants. A notable exception is Western Province; however, this is mostly due to small sample size (seven POs). Consistent with general assumptions, more urbanized provinces appear to have accrued more benefits from LPCB, but this can be attributed to the generally higher levels of existing capacity in those areas. Additionally, these results show that all cohorts and provinces report a net increase in perceived capacity. While IDF scores alone are not an objectively quantifiable measure of organizational capacity, they may indicate the extent to which an organization perceives it has changed in response to LPCB support. 37 Operational Capacity Of the 35 POs sampled for interviews, 86% reported an increase in general operational capacity due to LPCB. The most frequently cited effect of LPCB was the establishment of systems; POs across all cohorts and provinces, regardless of graduation status, cited the development and adherence to proper systems as one of the most significant impacts of LPCB. POs specifically cited M&E and financial reporting as the two most valuable systems established through LPCB. Those POs in rural areas and with lower relative capacity were more likely to report that prior to LPCB, such systems were not in existence. To that effect, the establishment and adherence to M&E and financial reporting systems in such POs may be directly attributed to LPCB inputs. Perhaps the most measurable outcome in this regard is the mere ability for POs of lower capacity to submit monthly reports to LPCB; five POs reported not tracking any data prior to participating in LPCB. Twenty-six percent of POs attributed improved ability to document and track HIV/AIDS activities specifically to service delivery grants, which required POs to report on activity-specific indicators on a monthly basis. For larger POs with relatively higher existing capacity, LPCB was reported to have influenced the refinement and sharpening of existing systems to become more streamlined and efficient. As opposed to larger POs, for whom training in M&E and financial reporting served to improve existing skillsets, smaller POs experienced greater gains (in absolute terms) in operational capacity from simply being able to track and report data where the capacity to do so previously was absent. Similarly fundamental to reported increases in operational capacity was the proliferation of individual skillsets in a variety of areas, including M&E, financial accounting, and program management. One of LPCB’s greatest reported impacts on day-to-day PO operations was found to be increased individual knowledge of how to do one’s job; an output of LPCB’s Objective 1, building the knowledge base of PO staff trained through LPCB was found to have directly contributed to improved organizational function. This increase in knowledge of job function was reported to have contributed to improved activity planning, better quality service delivery, and increased financial viability. An additional outcome with regard to increased individual skillsets was found to be improved staff morale; nine POs reported high staff motivation as a direct result of LPCB support. Staff motivation was similarly found to have impacted service delivery quality and efficiency, as six POs reported positive changes in program implementation due to improved understanding of discrete staff roles and responsibilities. Changes in operational capacity were found to be greatest among smaller POs, which were more likely to express that prior to LPCB, operations were highly disorganized. One reported challenge associated with increased individual staff capacity was the retention of skilled labor. Four POs reported having experienced staff turnover after their engagement with LPCB ended, as staff whose skillsets were bolstered with knowledge of cutting-edge best practices in organizational development left the PO for employment elsewhere. In two cases, these staff members were absorbed by an international donor agency. At an individual level, employee turnover stands as a testament to the success of LPCB in raising an individual’s capacity to a level at which s/he can obtain more fruitful employment. However, the cost of losing such staff from the perspective of the PO is significant, as the PO is left to fill a critical capacity gap. Seven POs reported finding it challenging to invest time and resources in the development of staff capacity, only to have trained staff recognize their ability to achieve a higher salary and leave the organization where it began. “Even if we die today, systems remain in place. Previous strategic plans were never used, just put on a shelf. But now we continue to use our plans and manuals all the time.” – Cohort 4, Eastern Province “Now we have knowledge and skills and confidence; no one can take that away from us.” – Cohort 2, Eastern Province 38 A significant factor in the reported increase in operational capacity was the procurement of office equipment from an LPCB commodity grant. Naturally, those POs applying for, and ultimately obtaining, commodity grants were those with lower relative existing capacity, as well as those in more rural areas. Of the 35 POs sampled, 54% received a commodity grant, to which 100% of POs directly attributed their increased organizational efficiency. These organizations were found to have made the greatest gains in terms of organizational productivity; several POs reported vast improvements in filing systems due to the acquisition of cabinets and shelving, which in turn enabled more effective and timely project reporting. One rural PO acquired a laptop computer – the first in the history of its existence – which enabled it to begin tracking activities, maintaining records, and communicating with other partners for the first time. After engagement with LPCB ended, 91% of POs sampled reported continued adherence to the systems put in place by LPCB. In 16 cases, POs adapted components of LPCB reporting protocols into their own routine operation, indicating the degree to which LPCB lessons were valued among POs, as well as the quality of LPCB inputs. In all cases, the most significant effect of LPCB in this area may be POs’ reported understanding of the value of having functioning systems in place: 10 POs explicitly stated that regardless of changes in staff turnover, systems that remain in place will continue to sustain PO operations. Management Capacity Of the 35 POs sampled, 54% reported an increase in management capacity during and after LPCB engagement. The most commonly cited LPCB input directing this change was the Leadership and Governance core training, which POs reported clearly delineated the roles of management staff and the Board of Directors. POs from rural areas were more likely to rate the impact of governance training as having played a critical role in its overall capacity development, citing the management challenges faced prior to LPCB. Specifically, 11 POs reported that poor understanding of the functions of a Board of Directors hindered organizational progress in some instances. Program staff of one PO stated that prior to LPCB, the Board tended to involve itself in day-to-day management decisions and unilaterally steer the direction of activity planning and implementation. Compounding this challenge was the fact that the founder of the organization, also head of the Board, was not open to the democratic process surrounding the IDF and strategic planning. However, after attending LPCB’s governance training and working with their assigned ODF, PO staff reported that the founder has embraced the true role of a Board member, and the Board is “functioning effectively” now that engagement with LPCB has ended. Other POs in more rural areas attributed decreased tension between the Board and management staff directly to LPCB. Improvements to organizational management is central to the development and sustainability of a PO, reported an ODF assigned to four POs in the Northern Province. According to him, the potential hindrance posed by an organization’s Board is greatest for small, rural POs, whose Boards are more likely to be comprised of individuals with limited knowledge and experience. Board members of some Northern Province POs also reportedly had expectations of personal or financial gain, and would often attempt to use PO resources for their own benefit. Despite the technical assistance provided through trainings and ODF support, these types of occurrences were cited as having significantly slowed the progress of organizational development experienced by the PO. The failure of LPCB to properly scrutinize the caliber and capacity of a PO’s Board upon entry to LPCB may have affected the degree to which POs achieved success over the life of the project; however, all rural POs sampled citing such challenges reported that “We already had capacity to deliver, so LPCB helped keep us on track with what we said we would do.” – Cohort 3, Lusaka Province “We were very disorganized before but are better now.” – Cohort 1, Lusaka Province “We developed more relevant strategic plans and made ourselves keep to it.” – Cohort 3, Lusaka Province 39 LPCB helped to mitigate pre-existing management challenges. On the other end of the spectrum, some larger POs with relatively higher capacity also experienced management challenges, particularly those that are branches of larger international NGOs. Grassroot Soccer and Lifeline Zambia, for example, are two such POs with an established international presence controlled by a Board largely external to its local Zambian activities. In the case of Grassroot Soccer, the organization’s governance assessed LPCB’s input as too minimal to merit continued effort and attention, and thus ended the partnership with LPCB. Program staff of Lifeline Zambia expressed uncertainty regarding the implications of future Board decisions on local chapters, but reported that overall, its board is effective. Because these types of organizations are not truly “local” partners, the degree to which LPCB was capable of effecting measurable change in their broader governance capacity may have been capped to some extent. Nonetheless, for POs across all cohorts and provinces sampled, management capacity was found to have increased at the local PO level. Technical Capacity Technical capacity among POs across all cohorts and provinces was reported to have increased at both the level of the organization and individual as a result of the Behavior Change Communication and HIV Technical Update trainings. Despite an admittedly absent focus on building technical capacity during the first two years of LPCB implementation, as reported by LPCB project staff, 12 POs sampled from Cohorts 1 and 2 reported an increase in capacity to reach target populations with evidence-based HIV prevention services. Later cohorts were found to have benefitted from the addition of HIV Technical Update training introduced beginning in Cohort 3, as 15 POs sampled from cohorts 3, 4 and 5 reported gaining and applying up-to-date knowledge in evidence-based HIV intervention strategies. The most frequently cited technical skills gained among Cohorts 1 and 2 included the knowledge and ability to target messaging to specific populations, whereas POs from Cohorts 3, 4 and 5 were more likely to cite a shift from mass prevention campaigns to small group-focused interventions. Eighty nine percent of POs reported the desire for additional, continuous follow-up training in technical competencies, indicating LPCB’s effect on the demand for technical skills at all levels of PO capacity. Of note, technical capacity was frequently cited to have been greatest for POs during their engagement with LPCB. Those POs desiring additional training in HIV-related technical areas often reported that in the absence of follow-up instruction, program staff were less likely to implement accurate or relevant activities in their communities after LPCB disengaged. Nine POs reported additional training was necessary due to the ever-changing face of the HIV/AIDS epidemic. The effects of increased technical capacity on service delivery are discussed under Key Question 4. Financial Viability All POs, regardless of province, cohort or graduation status, reported an increase in general financial viability as a direct result of LPCB. Financial viability was reported by POs in terms of three measures: change in overall funding levels, change in number of funding sources (beyond LPCB), and change in resource mobilization capacity (grant writing). Even those POs sampled who only participated (i.e. only ever received training) still attributed higher capacity to mobilize resources and write successful grants directly to LPCB inputs. Financial viability was found to have increased in several ways. First, and most directly, financial viability was “People are attracted to an organization that appears organized.” – CL, Lusaka Province 40 found to have increased as a result of improved resource mobilization capacity. Individuals who attended the Resource Mobilization core training from among the POs sampled reported improved proposal writing skills and greater confidence in writing successful proposals. POs also cited the mentorship of their CL/ODF as critical to the development of resource mobilization skills. These skills were found to have been further enhanced by applying for grants through LPCB; the mere practice of writing a proposal to conform to the rigid guidelines of USAID was cited as one of the most valuable components of the project, regardless of cohort or province. Secondly, 13 POs reported that networking with other POs has improved financial standing, as increased partnerships with other organizations has led to additional funding opportunities. The majority of these POs credited LPCB for initiating partnerships through designated networking opportunities throughout the life of the project; however, a number of POs reported having gone beyond LPCB to forge additional partnerships with non-LPCB organizations. Even if such partnerships did not immediately yield financial benefits, POs still reported increased financial viability as a result. Thirdly, eight POs reported being able to attract additional funders and partners by demonstrating M&E and financial reporting systems established and refined due to LPCB inputs. In several cases, POs reported being asked by a potential donor to provide evidence of a functioning financial accounting system. These POs attributed the ability to demonstrate functioning systems to donors directly to LPCB, which, in turn, helped to improve financial standing and donor confidence. Finally, six POs reported that simply having participated in LPCB was found to have attracted additional donors who may otherwise have bypassed their organization. This phenomenon was found to be due in part by LPCB referring additional partners to graduated POs upon completion of project objectives. Further, however, donor initiatives beyond LPCB were found to have taken notice of POs having undergone capacity building as part of LPCB. In fact, ZPI management staff indicated that a “bonus” effect of LPCB was ZPI’s preferential selection of POs known to have participated in LPCB, who were seen as more “trustworthy” and capable of absorbing funds than those who did not participate. The fact that roughly half of current ZPI-funded organizations took part in LPCB speaks to the overall success accrued by LPCB beneficiaries over the life of the project and beyond, as well as the demonstration effect of LPCB on the behavior of other donor-funded initiatives. This effect has additional implications for the sustainability of LPCB’s inputs and the broader donor community, discussed in further detail under Key Question 5. “Some [POs] just want money, but they need skills first.” – Cohort 2, Southern Province 41 Figure 4. Reported Changes in PO Financial Status Post-LPCB Of the POs sampled, 46% reported an increase in the number of funding sources since disengaging from LPCB. Twenty-three percent reported an increase in both number of sources and overall level of funding, while only nine percent reported an increase in funding level alone. Donor fatigue was the most commonly cited reason for stagnant funding levels post-LPCB. Twenty-three percent of POs also reported no change in finances post-LPCB; however, the majority of these respondents were part of Cohort 5, whose engagement with LPCB ended just prior to this evaluation. Further, 100% of these POs reported having recently submitted grant proposals with the expectation of securing additional funds/funders within the coming year. Women-led Organizations Of the POs sampled for this evaluation, LPCB was found to have affected both men and women-led organizations roughly equally. Table 3 below shows that both men and women-led organizations performed similarly in terms of successfully accessing an LPCB service delivery grant, as well as successfully completing the established requirements for grant implementation (graduation status). Table 6. Select PO Measures: Sampled Women-led POs vs. Other Type of PO % Received Service Delivery Grant % Graduated or Completed % Reporting IGA Women-focused or led (11) 64% 70% 55% Other (24) 58% 80% 22% Where the performance of men and women-led organizations appears to diverge relates to POs’ reported engagement in income-generating activities as a means of reducing donor dependence. Over half the number of women-led organizations sampled reported increasing financial viability through current, ongoing entrepreneurial activities, as opposed to less than a quarter of men-led organizations. IGA, a component of the LPCB training in resource mobilization, was also cited to have been explicitly encouraged by CLs/ODFs to reduce financial vulnerability. Women-led POs sampled from rural areas were more likely to raise income for their organizations through activities such as chicken-rearing and property rental, whereas those in more urban areas tended to raise funds through training consultancies. One rural women-led PO reported using the financial management skills attained through LPCB trainings to improve the management 23 23 9 46 0 5 10 15 20 25 30 35 40 45 50 No Change Increased Number of Sources and Funding Levels Increased Overall Funding Levels Increased Number of Funding Sources PERCENT (%) OF POs Financial Viability Post-LPCB 42 of payments collected from the rental of an accessory building built from scratch by its female members. Another rural women-led PO cited the skills gained from LPCB – specifically, the encouragement of small group-focused interventions – to establish a revolving fund for its members, whereby members funnel a portion of income earned from the fund’s microloans directly back into the organization. Given the increasing fatigue in the HIV/AIDS donor community, women-led POs cited the critical importance of shifting away from donor-dependent funding to sources over which the PO has greater control. In this way, women-led organizations sampled were found to have disproportionately benefitted from LPCB’s encouragement of IGA. However, this phenomenon may be more a function of the nature of women’s groups; women’s organizations participating in LPCB may have strong existing ties to community members and beneficiaries, making them more amenable to developing and maintaining the types of income-generating activities that require communal input and cooperation. It should also be noted that the decision to pursue IGA may hinge on an organization’s unique circumstances, including existing resources or IGA’s applicability to the organization’s stated mission or objectives. In this respect, IGA may not necessarily be viewed as an explicit “benefit” to an organization per se, but rather, a reflection of an organization’s resource management capability. The generalizability of conclusions regarding the extent to which success accrued to men and women-led organizations equally, however, is limited by the small sample size. Concurrent Capacity Building Support POs in urban areas were more likely to be receiving concurrent support from other initiatives, such as SHARe II and ZPI, as well as prior support from Capacity Leader organizations. One Cohort 1 PO reported that participation in LPCB allowed for smoother engagement with multiple concurrent USAID￾funded projects as a result of increased capacity to manage stringent reporting requirements. Sixty percent of POs with prior relationships with CLs reported benefitting from the value added by LPCB. Specifically, one Cohort 1 PO stated that the capacity building support provided by SAT prior was helpful, but that the addition of LPCB “accelerated” their growth. Ten POs also remarked that if they had not participated in LPCB, their capacity may still have grown, but not at the speed or level of quality provided with the support of LPCB. LPCB appeared to have been operating in an environment already conducive to capacity￾related interventions, aiding the extent to which LPCB inputs gained traction among beneficiary organizations, thereby contributing to PO success. While concurrent support from other projects and organizations was found to have impacted the rate at which a PO developed its capacity, POs reported that the greatest share of their success could be attributed to LPCB alone. Nine POs reported receiving concurrent capacity building support through other USAID-funded projects; 100% of those POs rated LPCB as the most effective model for capacity building by comparison. Reasons cited included the high level of PO buy-in, high quality workshops, and continued follow-up support provided by an ODF/CL. One PO remarked: [LPCB’s] Individualized approach was better than other approaches because it was not prescribed for us…Other projects would try to identify a problem area even if one didn’t exist. But LPCB was all about what we actually needed. Another PO stated: We’ve had many CB workshops in the past, but these were different. The combination of ODF, workshops, and grants was a uniquely powerful way of building capacity. If a [PO] didn’t get something out of LPCB, they must not have had the right intentions from the start. 43 An additional factor related to concurrent support was found to be the evolution of beneficiary expectations of technical assistance over the life of the project. POs receiving concurrent support presently or in the past were more likely to report higher initial expectations of LPCB support, as well as a shift in such expectations as the project ensued. Specifically, POs cited differences between the capacity building approaches of other projects and LPCB as a reason for adopting a new mindset organization development – one that emphasizes PO ownership of the OD process and its results. A change in attitudes toward OD exemplifies the success of LPCB to the extent that it strengthened the capacity of said POs to the point of acknowledging the value of local ownership. Potential for Organizations to Become Direct USAID Recipients Organizations with the highest operational, management, and technical capacity over the life of the project were Capacity Leader organizations, which were deemed as such precisely because of their existing capacity. These organizations were found to hold the greatest potential to become direct recipients of USAID funding, having undergone incremental refinements in their systems, processes, and overall capacity to sub-grant to smaller organizations over the life of LPCB. CLs notwithstanding, a number of POs demonstrated the capacity to manage funds soundly and in accordance with USAID standards by successfully attaining and implementing Innovative Grants. Having “passed the test” of financial accountability, these organizations were more likely to be located in urban areas and joining LPCB with higher overall levels of capacity than their smaller counterparts. Those six POs may be considered additionally capable of managing USAID expectations in that alignment with USAID objectives was considered a precondition for being awarded an Innovative Grant. Nevertheless, it should be noted that even these “highest-performing” POs requested further training in critical skills to continue their organizational growth, indicating an acknowledgment of residual capacity gaps. By the end of the project, the majority of POs have not yet reached a level at which management of direct USAID funding is possible; however, these organizations have still have exhibited measurable growth in overall capacity. High-performing (graduated) organizations in rural areas, in particular, were found to have demonstrated success not only in managing and accounting for LPCB funds, but in serving as skills training hubs for individuals and smaller organizations in their respective communities using resources provided to them through LPCB. These organizations may not necessarily aspire to rise to the level of a SAT or ZHECT, but still hold the potential to manage donor expectations and multiple concurrent activities. CONCLUSIONS LPCB considerably strengthened institutional and technical capacity of beneficiary organizations. For smaller POs, the largest impact was the creation of systematized operations and standards. For larger POs, existing systems were sharpened and refined. POs reported that increased technical capacity allowed them to improve the quality of HIV/AIDS service delivery. LPCB created demand for additional technical capacity among beneficiaries. All organizations experienced improved financial viability as a result of LPCB, even if only provided with training. Sampled women-led organizations in particular have benefitted from income￾generating activities. Capacity is inherently difficult to quantify. 44 LESSONS LEARNED LPCB benefitted from being implemented in an environment in which concurrent support for organization development was available. Because LPCB inputs were clearly distinguishable from those of other projects, beneficiaries could easily confirm the comparative advantages of LPCB’s comprehensive approach to organization development. The increase in demand for technical training exemplifies strengthened capacity to the extent that POs now “know what they don’t know”. PO capacity to identify and respond to gaps in organizational function demonstrates project success. 45 4: QUALITY HIV/AIDS SERVICE DELIVERY KQ 4: What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? FINDINGS The evaluation team used the following framework to answer this multi-tiered question: Did POs successfully increase the quality of their HIV/AIDS service delivery (where an increase in quality is defined as a reported improvement in the services delivered) as a result of LPCB activities? Did POs successfully increase the quantity of their HIV/AIDS service delivery (where an increase in quantity is defined as a reported increase in the number of beneficiaries served) as a result of LPCB activities? What were the strengths and weaknesses of LPCB’s model for improving HIV/AIDS service delivery through organizational development? Improvements in Quality In the absence of a PO service delivery quality assurance system, it is not possible to draw verifiable conclusions on the impact of LPCB on quality HIV/AIDS service delivery. The following findings and conclusions are, thus, based on the reports of partner organization staff. LPCB’s stated goal was “increased and better quality HIV/AIDS service delivery by LPCB Partner Organizations in Zambia. The project utilized a multifaceted approach to improving the quality of PO service delivery, including the provision of training, technical assistance, and grant opportunities. After training the first two cohorts, LPCB project staff incorporated feedback from POs for more technical training that was directly applicable to HIV/AIDS service delivery. Consequently, the project added three trainings that focused on building POs’ technical capacity in the HIV/AIDS prevention sector. The next phase of LPCB was geared at improving quality was to provide grants to POs, which were designed to facilitate next steps for POs that completed the training. The key assumption made by the project was that improved organizational development would lead to higher quality service delivery outputs on the part of the POs. Key informant interviews (KIIs) revealed that POs frequently conflated OD and SD, and had no clear metrics to measure improved quality in service delivery. The evaluation team found that PO attitudes about quality in HIV/AIDS service delivery changed significantly as a result of the organizational capacity building. POs reported that the trainings helped them to internalize the importance/value of high quality service delivery. One key informant noted that “the training has helped us define and achieve our goals, objectives, strategies and activities. Now we have more purposeful goals including delivery of more services and better quality services.” LPCB successfully imparted key PEPFAR focal areas to POs, leading to a shift from standardized mass campaigns to targeted interventions in small group settings and improved PO knowledge of how to target beneficiaries with evidence-based interventions. POs reported a significant improvement in the actual quality of services that they were able to offer as a result of the project. Seventy-one percent of POs interviewed stated that the quality of their organization’s 46 service delivery had been positively affected by participation in LPCB. Specifically/overall, they noted improvements in organizational ability to: Identify gaps in service delivery Maintain updated financial records and practice good financial management Develop a culture of documentation Create and maintain sound organizational systems and standard operating procedures Monitor activities and track success against indicators and targets In terms of service delivery, POs reported the following outcomes: Increased community buy-in, as a result of targeted community engagement efforts Increased access to donor funding Increased access to partnering and networking opportunities Improved quality assurance procedure for service delivery However, although POs reported both strong indications of improved organizational development and several positive outcomes pertaining to SD, the direct connection/overlap with actual service delivery achievements was found to be peripheral, rather than a one-to-one relationship. Like the LPCB project itself, POs did not differentiate between improved capacity or ability to deliver quality services and the actual delivery of said services to beneficiaries. This missing link is reflected in the type and primary focus of the trainings LPCB provided to the POs. Seven out of nine of the core training electives focused on organizational development principles, while only two technical trainings were directly relevant to HIV/AIDS service delivery. LPCB’s grant component, provided sums of money to organizations to execute their high quality delivery concepts, but the technical support linking organizational development to the higher level object – service delivery – was lacking. One key informant noted that “LPCB as a project, would have been strengthened if LPCB was first and foremost designed to meet PO organizational capacity needs, which at its core is focused on quality and quantity of health service delivery to recipients.” An example typifying the missing link between OD and SD in self-reported PO successes was one KI’s description of success in improving SD quality by “identify(ing) gaps (in the organization’s core competencies) and hire skilled employees to fill them.” Here, the KI represents what is actually an achievement of the project’s intermediary goal (to build organizational capacity), as an improvement in quality of SD. However, there is no link to justify the assumption that organizations would be able to make the jump from hiring skilled employees to effectively utilizing those employees in a way that translated to higher quality service delivery. To the extent that organizations were able to independently recognize this link, LPCB was able to maximize gains and establish the connection between OD and SD. As this link was not explicitly made, either in the project logic or to the POs, the observed increase in quality is difficult to attribute directly to LPCB. Improvements in Quantity Due to the absence of reliable service delivery baseline data (indeed, a number of POs had no mechanism for tracking data prior to enrolment in LPCB), a thorough analysis of LPCB’s effect on the quantity of HIV services provided by POs is not possible. PO Performance data post-LPCB is similarly unavailable for the purposes of before and after comparison; hence, multi-year trends and the sustainability of PO results cannot be determined. Partial aggregate data on the main PEPFAR prevention indicator tracked (P8.1) for 27 recipients of service delivery grants, however, was made available to the evaluation team and is summarized in Table 4 below, along with additional descriptive measures. Partner organizations are listed in 47 descending order of the number of people reported to have been reached with preventive services in accordance with minimum PEPFAR quality standards during the reporting period. Table 7. Aggregate P8.1 Reporting for Select POs PARTNER ORGANIZATION P8.1 REPORTED PROVINCE COHORT GRANT VALUE (USD) IDF SCORE CHANGE (%) PO TYPE Youth Development Organization (YDO) – Choma 13880 Southern 1 115739 42 CBO Chilanga Youth Awake - Lusaka 13208 Lusaka 1 63653 20 CBO Action for Positive Change (APC) - Chipata 13118 Eastern 1 63777 36 CBO Luapula Families in Distress (LUFAID) - Mansa 9591 Luapula 1 68482 13 NGO Groups Focused Consultants - Mansa 8883 Luapula 3 40116 -- NGO Bridge of Hope Foundation - Chipata 8429 Eastern 2 74230 70 NGO Contact Trust Youth Association (CTYA) - Livingstone 7848 Southern 1 106162 32 CBO Nchelenge Interdenominational Youth Learning Sharing Initiative Group - Nchelenge 7451 Luapula 2 66802 24 FBO Great Commission for People Development and Orphans (GCPDO) – Chipata 7008 Eastern 3 24617 -- FBO Children In Distress Project (CINDI) – Kitwe 5596 Copperbelt 2 105653 4 CBO Kafue Child Development Agency (KCDA) - Kafue 5501 Lusaka 1 104750 4 NGO Kasama Christian Community Care – Kasama 5209 Northern 4 41737 -- FBO Katete District Women Development Association - Katete 4951 Eastern 2 50307 -13 CBO Tulipamo AIDS Support – Kapiri Mposhi 4680 Central 3 44173 6 NGO Anti-AIDS Teachers Association of Zambia (AATAZ) - Lusaka 4174 Lusaka 1 41598 24 NGO Mthuzi Development Foundation (MDF) - Chipata 4018 Eastern 1 33449 31 CBO Community Based TB/HIV/AIDS Organization (CBTO) - Lusaka 3803 Lusaka 2 93718 22 CBO Luapula Foundation - Mansa 3265 Luapula 2 255434 -8 NGO Rise Community Aid Program (RICAP) - Kafue 3220 Lusaka 2 41274 23 CBO Adolescent Reproductive Health Advocates (ARHA) - Mongu 2977 Western 3 46620 -17 CBO Sachibondu HIV and AIDS Prevention and Education (SHAPE) - Mwinilunga 2566 Northwestern 4 38829 20 FBO Afya Mzuri - Lusaka 2481 Lusaka 4 206772 -21 NGO Kwenuha Women Association - Livingstone 1667 Southern 1 42604 25 FBO Chisomo Home Based Care - Chipata 899 Eastern 2 51345 68 FBO Expanded Church Response (ECR) - Lusaka 742 Lusaka 4 129413 -- FBO Solwezi Youth Alive - Solwezi 652 Northwestern 4 38275 13 NGO Network Of Zambian People Living with HIV and AIDS - Mansa 383 Luapula 2 51918 22 CBO 48 It is important to reinforce that it is not possible to attribute results directly to LPCB, especially in the absence of baseline and control group data; “high-performing” POs may well have been high performers before their engagement with LPCB. Nonetheless, the above evidence shows those POs that are highest￾performing with respect to reported volume of people receiving preventive interventions are more likely to have been enrolled in earlier cohorts. Interestingly, however, there does not appear to be an immediate correlation between PEPFAR reporting volume and urban/rural location. Further, the evaluation team found no statistical correlation between PEPFAR reporting, amount of funding received from LPCB, or IDF score change. Consequently, based on the data provided, neither the amount of LPCB financial input (service delivery grant value) nor POs’ perceived change in capacity were related to the number of people reached with preventive services. The relationship between volume of service delivery and graduation status is similarly tenuous, as the third, fourth, and fifth-ranked POs were judged to have only “completed” their service delivery grants, despite reporting high quantities of services delivered. It appears generally true that larger, high capacity organizations achieved higher numbers, though notable exceptions include AATAZ and Afya Mzuri. There is also no discernible difference between PO performance by type of organization (NGO, FBO, or CBO), indicating that the main drivers of PEPFAR reporting performance are more likely to be unobservable variables independent of those listed above. Triangulated with qualitative interviews, these findings reveal noteworthy facets of LPCB’s impact on service delivery generally. LPCB project inputs were found to be largely oriented toward the development of institutional capacity in POs, whereas the indicators used to measure LPCB success were chiefly focused on the volume of services provided. The misalignment of project inputs and results indicators underscores the project’s apparent reliance on weak assumptions regarding the effects of strengthened technical capacity on service delivery. Above variables of geography, grant amount, and perceived change in capacity, the most significant determinants of service delivery quantity were found to be related to an organization’s target population and individual mission. For example, POs whose activities were already aligned with PEPFAR indicators reported the ability to readily document their impact, particularly if the PO had a large target population or catchment area. Other POs, however, have more focused target populations (e.g. commercial sex workers); thus, their ability to report high numbers of services provided is limited, despite reporting significant gains in institutional and technical capacity. Accordingly, the inability to draw conclusive links between LPCB monetary input and the end result of service delivery reinforces qualitative findings that different POs harnessed capacity growth in different ways. While some POs reported smoothly translating capacity growth into increased services, others reported focusing on quality, while others reported a sustained focus on building internal processes and financial resources. Thus; organizational development was found to have affected multiple areas of PO function, not all of which relate purely to service delivery quantity. Ultimately, in terms of reporting toward PEPFAR targets, low numbers do not necessarily mean low performance or low capacity. As evidenced by a lack of uniform trend among POs reporting on prevention indicators, POs appear to have been incentivized to reach targets unique to their individual core missions rather than the universal goal of increased quantity. 49 Figure 5. Reported Changes in Service Delivery among Sampled POs Fifty-three percent POs surveyed reported in an increase in the quantity of their HIV/AIDS service delivery, primarily as a function of expanding the scope of their target population in response to improved technical skill sets and heightened awareness of evidence-based communication strategies. POs reported that newly￾breached populations tended to be comprised of religious factions previously resistant to collaborating with HIV-related organizations. Despite these perceived gains, POs reported feeling pressured to meet quantity targets, even after these targets were revised down. “A big emphasis of the LPCB Project was its focus on meeting quantity targets…We felt this might hurt the quality of our services,” noted one PO. While the project’s emphasis on high quality service delivery was met with significant levels of uptake, this appears to have had unintended consequences on the sheer volume of HIV/AIDS service delivery by POs. One key informant characterized the shift, saying “we were so focused on quantity before, but now we stress quality above all else.” The seemingly disparate goals of LPCB, a dual focus on quantity and quality, created a mismatch between the project’s indicators, (focused on quantity), and the project’s perceived and stated goals, (focused on quality). As a result, several POs reported a deliberate decision to prioritize quantity, even at the expense of reaching fewer beneficiaries. These POs pointed to the fact that improving quality, in addition to the organizational development skills provided by LPCB, would aid their long term goals of expanding donor networks and remaining sustainable. Strengths and Weaknesses of the LPCB Model The crux of LPCB’s shortcomings stemmed from the lack of strategic planning at the project design and implementation stages. The LPCB results framework, approved approximately one year after project inception, was not sufficiently used as a point of reference during activity development and implementation, as reported by project staff. Thus, the crucial process of scrutinizing a development hypothesis and linking each input to intermediate results and development objectives did not occur. LPCB assumed a logical progression from improved organizational capacity to increased quantity and quality of HIV/AIDS services without articulating the predicted linkages in a development hypothesis. Additionally, the indicators POs reported on monthly, while more amenable to individualized PO activities than PEPFAR indicators, assumed that a high volume of service delivery was a universal goal for the organizations working with the project. In fact, some POs were so specialized in providing services to a particular target population (e.g. sex workers, youth) or so constrained by rural locations, that numerical targets were unrealistic from the outset. Problematic funding delays were also found to have negatively impacted service delivery and PO-beneficiary relationships. 11% 29% 53% 18% PO Service Delivery Post-LPCB Increased Service Quantity Increased Service Quality Increased both Quantity and Quality Reported Improvement Unrelated to Quantity or Quality 50 A notable strength of LPCB was its use of a cooperative agreement, rather than a contract. This allowed the project to be flexible in modifying its activities and remaining responsive to challenges and lessons learned throughout the intervention. LPCB also excelled at incorporating PO feedback and being responsive to PO requests. The project incorporated PO demand for technical assistance into successive iterations of trainings, as well as adding grants to facilitate service delivery after recognizing a missing link between the OD and SD goals of the project. While only offered to six high-performing POs, these “bonus” innovative grants were found to have directly contributed to service delivery improvements. For example, one PO used its grant to design a highly successful VCT QA tool for clinical settings, which was subsequently widely adopted by POs across provinces. Similarly, “Youth Friendly Corners” in clinics were established from an innovative grant, linking the PO’s VCT services with clinic treatment for beneficiaries referred for care. Both homegrown initiatives flourished because of the project’s flexibility in creating the grant, as well as the grant’s design, which was designed to incentivize quality and PO ownership of SD activities. CONCLUSIONS The quality of services delivered by POs cannot be objectively verified due to the absence of a quality assurance system. The project successfully increased POs capacity to deliver higher quality HIV/AIDS services, while increasing the actual delivery of services to a lesser, and indirect, extent. Organizations that independently recognized and maximized the linkage between organization development and service delivery, such as recipients of innovative grants, fulfilled the higher level service delivery oriented goal of the project. Innovative grants were found to have successfully influenced POs to conceptualize and carry out service delivery improvements in their communities, reportedly contributing to increased service delivery quality. The tenuous link between the LPCB goal of increased and improved service delivery and its primarily organizational development-focused inputs created confusion among POs about the purpose of the project The link between OD and SD exists, but LPCB’s lack of strategic planning in PMP development at the project’s inception diluted its effects and left the project’s potential to establish a valuable theory of change unfulfilled. This has implications for the replicability of LPCB’s successes. Some POs missed the opportunity to translate their newly developed organizational capacity to service delivery because they understood LPCB to be primarily a capacity building project, and were not equipped with the technical skills/clear guidance on how to consistently improve quality as a result of improved organizational capacity. Benefits of LPCB accrue to many activity areas outside of HIV that may not be measurable using PEPFAR or other service delivery-oriented measures. LESSONS LEARNED Increased capacity may not immediately translate into gains in service delivery. Organizations can choose to apply improved capacity toward a number of organizational areas not necessarily aligned to service delivery-oriented goals. Achievements in both increased capacity and service delivery are possible, but a project attempting to simultaneously focus on capacity building as a means to an end (service delivery) and an end in and of itself (organization development) may not be able to achieve both outcomes equally. 51 5: SUSTAINABILITY OF RESULTS KQ 5: What are the prospects for the sustainability of the capacity building results for the local partner organizations and institution strengthening providers? FINDINGS Perhaps the most critical component of LPCB, emphasized from inception to project close-out, is the sustainability of results. Indeed, POs, CLs and ODFs consistently reported the degree to which LPCB stressed sustainability in each phase of implementation under the notion that capacity building is only as effective as it is sustainable. To that end, the findings presented throughout this report suggest that one of the greatest prospects for sustainability is the increase in individual knowledge and the expansion of skill sets relevant to technical areas. Of the 35 POs sampled, 60% reported that the one project component enabling organizations to remain sustainable in the future was training. As discussed earlier, training was highly valued among all POs, regardless of graduation status, cohort, or province. Similarly, 34% of sampled POs reported the establishment of systems as critical to organizational sustainability. Systems that remain, these POs noted, enable organizations to continue to function regardless of management changes of staff turnover. Despite gains in organizational capacity over the life of the project, the sustainability of results was widely reported to be threatened by the absence of follow-on training, particularly in technical areas. The emergence of widespread demand for technical skills – and organizational development, generally – is indicative of one of the project’s most significant accomplishments: the creation of demand for capacity. Nearly all POs interviewed expressed the need for additional capacity in order to reach their organizational objectives. Many of these POs also reported a newfound acknowledgment for the value of organization development and the significance of its process facilitation. Increased value for OD was additionally found to have kindled interest among beneficiaries in seeking higher education in competencies related to those taught by LPCB; one Cohort 1 PO reported that in response to assistance received through LPCB, one staff member left the organization to pursue an advanced degree in organization development skills. ODFs interviewed reported witnessing the same phenomenon occur in POs across different provinces. Coupled with reports of PO staff attrition to organizations offering higher salaries, this trend exemplifies both the advantages and inherent risks of building individual capacity. On one hand, PO sustainability is threatened by the temptation for individuals with strengthened capacity to seek more profitable employment elsewhere. On the other hand, LPCB and other capacity building projects may be credited for having improved employment prospects for individuals who may not otherwise have been exposed to resources for improved capacity. One of LPCB’s activities undertaken for the explicit purpose of sustaining results was the provision of networking opportunities, which were found to have benefitted POs in a number of ways. By providing a platform for POs to share their experiences with one another, LPCB’s networking events, meetings, forums, and consortia established a system for continuous knowledge sharing and collaboration. This platform was found to have produced additional effects beyond the bounds of dedicated project activities; LPCB served as a catalyst in the spurring the development of spinoff collaborations among POs wishing to broaden their collective community impacts. The continued success of PO-led collaborations such as the Civil Society Framework for Responding to HIV, TB, and Malaria in Zambia (CSF) and the Local NGO Directors’ Forum may be attributed to some extent to support from LPCB. Of the POs sampled for interview, those who reported harnessing networking opportunities also attributed improved financial viability to the strength of partnerships formed with other LPCB POs, as well as some organizations unaffiliated with LPCB. Thus, networking and collaboration – either directly promoted or indirectly inspired 52 by LPCB – support the sustainability of project results to the extent that POs continue to reap the benefits of collaborative knowledge exchange and funding opportunities. As reported earlier, the explicit encouragement of POs to train beneficiaries in evidence-based innovations using skills acquired from LPCB core trainings was a particularly effective project activity. LPCB provided training directly to 1,915 individuals, while POs disseminated training to an additional 7,022 community beneficiaries, demonstrating the capacity for POs to multiply the results of LPCB through peer education. However, PO site visits revealed yet broader-reaching project externalities yielding significant implications for sustainability of results beyond LPCB’s inputs. Eight POs reported engaging community stakeholders in institutional capacity training activities separate and distinct from those encouraged at the behest of LPCB. Of note, the majority of these were smaller POs located in rural areas, who began administering training for other organizations in the same core competencies imparted to them by LPCB. Two years post-LPCB, one PO reported providing continuous financial management training to the community’s chieftain. The same PO also offers regular IT and computer skills training to youth in the community using equipment provided through LPCB. Other POs, effectively serving as “mini-CLs”, reported that engagement with LPCB raised their visibility and credibility in their respective communities, making it easier to spread knowledge. Strengthened credibility also had the reported effect of attracting other community organizations in search of specific skills to LPCB POs for training; as one PO noted, “Now, other organizations approach us to meet their capacity [needs]. We are leaders in our community.” In addition to benefits accrued directly to POs, LPCB’s inputs were found to have affected the outlook of the external donor environment to some extent. It was reported that international agency World Vision, a longtime actor in the Zambian HIV/AIDS arena, approached LPCB with the purpose of adopting a similar model for its own capacity building project. The demonstration effect of LPCB suggests further project success in inducing a shift in private sector priorities toward a focus on institutional capacity for the achievement of results. To the extent that the wider donor community continues to imitate LPCB’s approach to capacity building, the ability of LPCB to influence the activities of external donors ensures that project results are sustained in an environment ever-conducive to the prioritization of organization development. The IODF was established as a means of supporting a cadre of Zambian OD professionals as a tangible legacy of LPCB. LPCB has indeed succeeded in building this cadre and providing a platform through which ODFs can effectively advocate for OD. However, the long-term sustainability of the IODF remains difficult to ascertain. According to one ODF, the IODF’s true test will be what happens in the months following LPCB’s close-out, noting that all IODF activities thus far have transpired with the direct support of LPCB. This ODF also reported the potential for the IODF to become analogous to a consulting firm wherein members compete with one another for individual paid consultancies. Further, the IODF is challenged in its ability to secure funds. Recent partnerships were reported to have increased the potential for IODF viability, but the IODF continues to operate in an environment in which funding opportunities specifically for OD remain scarce. Additionally, four of the POs sampled reported being unable to pay for the services of a former ODF. While it is also apparent that demand for OD has grown in recent years – demonstrated by increased private sector investment (World Vision) and donor-funded projects with capacity building components – organizations in greatest need of organizational development are unlikely to possess the capacity to pay for professional services. CONCLUSIONS Best prospects for PO sustainability include networking opportunities and the development of demonstrable systems and skills, which hinge on the successful provision of technical refresher trainings. 53 The multiplier effect of PO training provision to community stakeholders demonstrates the potential for added sustainability through project externalities. Some POs are effectively serving as “mini-CLs” as a result of LPCB. Private sector donors have begun to imitate LPCB’s design and approach in response to the visibility of LPCB achievements. LPCB has succeeded in building a cadre of OD professionals through the IODF, but its sustainability is challenged by a general shortage of funding opportunities. LESSONS LEARNED Relatively small inputs in organization development can yield large, sustained outcomes. Organizations that feel empowered by strengthened capacity will readily share their knowledge with community beneficiaries even in the absence of direct encouragement. Visibility – of the project, its results, and its beneficiaries – may be one of the key drivers of LPCB sustainability. 54 RECOMMENDATIONS Future capacity building projects should set project targets based on annual LPCB results. Targets should be kept realistic and account for organizations of lower-than-predicted capacity. Establish gender-specific targets that incentivize all POs, not simply those that are women-focused, to prioritize gender issues in delivering services. Establish indicators for measuring annual organizational growth in addition to IDF scores. Tracking such data will allow for more substantive conclusions to be drawn between organization development and service delivery. Useful indicators for measuring organizational phenomena over time may include: o Staff size o Number of staff who left organization within past year o Number of staff trained in quality HIV/AIDS service delivery o Frequency of IDF administration o Total number of beneficiaries served o Frequency of staff/board meetings o Number of partnerships formed with other organizations o Number of donors o Number of volunteers o Frequency of District AIDS Task Force meeting attendance o Frequency of District AIDS Task Force report submission Offer POs the resources to provide financial incentives (e.g. stipends, meals, transportation) with which to motivate volunteers. Increased volunteer motivation will help to mitigate challenges with volunteer retention and aid uninterrupted service delivery. Increase homogeneity of cohort composition by grouping together organizations of similar levels of capacity to maximize inter-organizational knowledge exchange. Enroll POs into larger cohorts that do not overlap. Allow for the completion of all project phases of one cohort before commencing support for a second cohort. While this may lengthen the overall number of project years, it is designed to reduce the potential for implementation challenges and strain on project staff. Establish two separate training tracks customized for organizations of relatively higher and lower capacities. Trainings developed for lower capacity POs should focus on organizational development basics, such as the establishment and maintenance of reporting systems. Advanced track trainings should focus on the refinement of existing systems and conforming to global best practices. Extend the minimum period of performance for service delivery grants to two years. This allows for more reliable tracking of the sustainability of capacity gains over a longer period of time. Increased duration of grant implementation provides POs the added benefit of documenting community impact over a longer period, reinforcing the significance of functioning M&E systems. Conduct baseline and end line capacity assessments modeled after the post-graduation assessment, in addition to administering the IDF, to better measure organizational growth over the life of the project. Assign a Capacity Leader Organization to each enrolled PO. Allow the IODF to apply as a Capacity Leader and assign it to POs with prior relationships with other CLs. This allows all POs to benefit from an organization’s resource pool and reduces potential biases CLs may already have for POs provided support external to LPCB. Establish training modules focused exclusively on training in entrepreneurship and IGA as a means of strengthening financial viability, in addition to grant writing workshops. Offer periodic refresher courses in technical service delivery and evidence-based interventions to 55 address high demand for technical skills. Prior to enrolment, conduct baseline capacity assessment for each PO using standardized indicators to measure internal organizational phenomenon, such as staff turnover, frequency of Board meetings, number of funding sources, etc. This will stand as an additional baseline measure for capacity that can be repeated over the life of a PO’s engagement, apart from the IDF self￾assessment. Establish a quality assurance system to verify the reported results of POs. Quality assurance checks, performed by CLs or ODFs, may be used to verify that PO service delivery meets internationally￾recognized quality standards and that PO monthly reports are completed accurately. Mandate the development of a Results Framework and a clear development hypothesis for any future capacity building projects. Future projects should provide early and comprehensive technical support in HIV/AIDS and the execution of high quality service delivery, as sustainable PO capacity will require continued technical support to build upon the early foundations of LPCB Reconcile high-level objectives with a process-oriented approach by balancing the focus of project activities between the goals of strengthening institutional and technical capacity (i.e. adding additional technical trainings). Scale up the provision of innovative grants to POs able to demonstrate activities linking OD to service delivery, particularly smaller POs, as this represents a direct investment in improved service delivery quality. Further, provide financial incentives for POs to link OD to SD through the inclusion of volunteer stipends as a component of innovative grants. Follow up capacity building projects should incorporate continuous follow-on training in technical competencies. Harness the potential for POs to serve as mini-CLs by incorporating training of trainer (ToT) opportunities into program design. This will, in turn, increase community buy-in to capacity building results. Raise visibility of LPCB and its achievements through a communication campaign targeting private sector actors throughout Zambia in order to further stimulate donor behavior change. Incorporate the IODF into the next capacity building project as a Capacity Leader Organization, which will raise its visibility among other participating organizations and increase its potential to secure reliable funding. 56 ANNEXES 57 ANNEX I: EVALUATION STATEMENT OF WORK Project Location Country-wide – Zambia Project Name Local Partners Capacity Building (LPCB) Project Project reference number 611-A-00-08-00005-00 under the Capable Partners Leader Award Number HFP-A-00-03-0002 Project budget $49,080,196 Funding sources PEPFAR/USAID Project duration May 19, 2008 to June 30, 2013 Implementing partners FHI 360 (formerly Academy for Educational Development) in partnership with Management Systems International Evaluation Type End of Project Performance Evaluation C.1 BACKGROUND In 2007, when Local Partners Capacity Building (LPCB) project was designed, Zambia had approximately 1.1 million people living with HIV/AIDS and a HIV prevalence rate among the adult population estimated at 17%. The United States Agency for International Development (USAID), through the President’s Emergency Plan for AIDS Relief (PEPFAR) and other donors were responding with increasingly large amounts of funding for HIV/AIDS prevention, care and treatment. The United States government, in close coordination with the Government of the Republic of Zambia (GRZ), was rapidly expanding programs and engaging new partners in the country. Engagement of local organizations was essential to reaching PEPFAR targets because of the organization’s unique understanding of the needs of their communities and how to adequately and innovatively address those needs. The massive scaling up of services required local organizations to absorb significant funding more effectively. In the PEPFAR context, this meant that local organizations had to learn to account accurately for financial spending and targeted results. Effective scale-up also required strengthening organizational systems and technical HIV/AIDS capacities, which many of the Zambian organizations lacked. Many Zambian organizations that were effective at reaching and working with communities or providing community-based services were not effective in operational, technical, and financial management. These organizations had limited capacity in work planning, strategic vision, human resource development/management, monitoring and evaluation, and reporting results. Furthermore, small community-based organizations receiving PEPFAR funding tended to lack HIV/AIDS technical capacity. In response to the critical need for more capable local organizations and to build the capacity of currently funded local organizations, USAID/Zambia designed LPCB. The main objective of LPCB was to improve the management, financial, technical, and monitoring and evaluation (M&E) capacities of targeted Zambian organizations, including local non-governmental organizations (NGOs), faith-based organizations (FBOs), and community based organizations (CBOs) to support their increased engagement and performance in the delivery of quality HIV/AIDS services in Zambia. To ensure sustained institutional capacity building within the country, USAID/Zambia included a complementary component of the design that would develop the capacity and expertise of select intermediary support organizations and individuals that would become sustainable entities working directly with local organizations as institutional strengthening service providers. C.2 PROJECT DESCRIPTION In May 2008 through a Cooperative Agreement under the Capable Partners Leader with Associates Award, LPCB was awarded to the former Academy for Educational Development (AED) (now under FHI 360) in partnership with Management Systems International (MSI). The purpose of LPCB is to improve the overall capacity of selected local NGOs, FBOs, and CBOs, collectively known as Partner Organizations (POs), to ensure efficient and effective expansion of their programs so that quality HIV/AIDS services in Zambia are delivered. LPCB has a life-of-project amount of $49,080,196. Since the beginning of the program, LPCB has been the flagship USAID/Zambia capacity building program. LPCB helps strengthen the management, financial, technical, and M&E capacities of Zambian organizations. It also supports a number of intermediary support organizations and individuals to develop the capabilities and expertise of local organizations. LPCB was designed under and contributes to the USAID/Zambia 2004-2010 strategic objective of Reduced Impact of HIV/AIDS through a Multisectoral Response, and associated intermediate results (see Annex 1). In addition, LPCB contributes directly to Zambia’s annual PEPFAR prevention, care and systems strengthening targets. As USAID and PEPFAR place greater emphasis on country ownership, LPCB’s visibility has been elevated in its capacity building models. For the full project description, please refer to Annex II. 58 C.3 PURPOSE AND USE OF THE EVALUATION The purpose of this evaluation is to assess program effectiveness and efficiency in order to improve future programming in strengthening local organizational capacity. For the purpose of this evaluation, the following definitions will apply; an activity is relevant if it is a required and necessary input for achieving an identified and clearly defined development outcome; an activity is effective if it produces the desired result, or development outcome it was designed to produce; an activity is efficient if it achieves its intended results within the allotted time and financial parameters; sustainability is achieved when host country partners and beneficiaries are empowered to take ownership of development processes, including financing, and maintain project results and impacts beyond the life of the USAID project. The overall objectives of the evaluation are: 1. To assess the extent to which project objectives, targets, outputs and expected results were achieved and or exceeded (performance); 2. To assess the effectiveness and potential impact and/ or sustainability of project activities and capacity building model(s) and approaches on the institutional and technical capacity of local organizations and subsequent expansion of quality community-based HIV/AIDS services. In particular, the evaluation will examine the extent to which the project succeeded in: a. Increasing the institutional capacity and sustainability of local partner organizations to efficiently manage funds and programs; b. Improving the technical capacities of local partner organizations to effectively implement and expand quality HIV/AIDS prevention, care and support activities and services; and c. Establishing a sustainable marketable cadre of experts and local intermediary organizations as competent institution strengthening service providers. 3. To inform the USAID/Zambia about future similar local capacity development activity designs and identify promising and competent local partners (replicability). The evaluation will also identify and document changes in LPCB’s model and activities since the Mid-Term Assessment (e.g. additional means for measuring organizational capacity, development and implementation of graduation plans and number of organizations that have graduated). It is anticipated that the evaluation will be useful to multiple audiences. The following list represents the intended use of the evaluation for some stakeholders: USAID/Zambia: To identify effective practices and areas for improvement and to inform the design and implementation of future interventions in building local organizational capacity in Zambia. To identify promising and competent local partner organizations among the POs and institution strengthening providers that could become viable direct local partners of USAID. USAID/Global Health Bureau, PEPFAR: To demonstrate the effectiveness and efficiency of engaging and strengthening the capacity of local organizations in expanding the delivery of quality HIV/AIDS services. USAID/Policy Planning and Learning Bureau: To suggest a model for evaluating capacity building programs. Government of the Republic of Zambia, and other donors: To demonstrate the effectiveness of strengthening local organizational capacity in expanding the delivery of HIV/AIDS services. LPCB, POs, Institutional strengthening providers: To learn their strengths and weaknesses and adjust their technical approaches for future projects accordingly. USAID/Zambia will use findings from the evaluation to inform the design of future projects and publications on capacity building interventions for local organizations. The report will be disseminated widely with relevant stakeholders and project beneficiaries as well as submitted to the Development Exchange Clearing House (DEC). C.4 EVALUATION QUESTIONS To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? Assess whether the activity managed to achieve the planned results focusing on quality/quantity of outputs for this activity. Assess the factors that facilitated or inhibited the achievement of these results. To what extent were the project design, implementation, and management effective and why? 59 Assess the best practices and lessons learned during each of the phases. Indicate any changes that occurred during implementation of this activity, both the external environment and or internal to the activity, in the evaluation report especially where they may have had a bearing on activity outputs and outcomes. Assess the effectiveness of the tools used to track changes in organizational capacity and whether these were good predictors of organizational success. Assess the effectiveness and efficiency of the institutional strengthening providers and areas of comparative advantage To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? Assess whether these changes or outcomes were comparable across cohorts and also across the provinces. Assess whether the successes accrued equally to men or women-led organizations. Compare changes that occurred during the period when the organizations were receiving LPCB support and those that happened after their support had ended. Indicate other concurrent organizational strengthening support that the organizations received with the LPCB activity especially where this may have had a bearing on organizational success. Assess the potential of the organizations moving on to become direct recipients of USAID funding. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? C.5 EVALUATION DESIGN AND METHODOLOGY C.5.1 Evaluation Design The evaluation will be carried out in Zambia by an independent evaluation team using a combination of qualitative and quantitative methods. Given that a counterfactual (control or comparison group) of local organizations was not established at the beginning of the project, USAID anticipates use of a non-experimental evaluation design for this evaluation. The project, however, collected some baseline information on organizations and tracked their progression during the interventions. Therefore, where possible, before and after comparisons to assess some of the changes in institutional and organizational capacity will be made. Contractors are requested to come up with creative and new ways to assess the capacity of these organizations, which may require reconstructing the baseline and designing new tools to assess in a more rigorous way the outcomes of the capacity building interventions. Contractors are required to elaborate a detailed evaluation design and methodology as part of their work plan. C.5.2 Data Collection Methodology The contractor shall sample 6 Partner Organizations (PO) from each cohort under the LPCB project for a total of 30 organizations. Of the Institutional Strengthening Providers (ISPs), the contractor shall sample 3 Capacity Leaders and 2 Organizational Development Facilitators. Therefore, the total sample size will be 35 organizations. At least 30% of the sampled organizations must be from less urbanized provinces including Northern, Luapula, Northwestern, Western and Eastern Provinces. The evaluation team shall provide a more detailed description of the proposed methodology for carrying out the work and how data quality will be ensured as part of their work plan. The methodology shall comprise a mix of approaches and tools appropriate to the evaluation’s research questions. These tools may include a combination of the following: Review of relevant LPCB project documents. USAID will avail to the evaluation team the following project documents and monitoring reports: a) Agreements, annual work plans and PMPs b) Quarterly progress reports as submitted to the Agreement Officer’s Representative (AOR) c) PEPFAR Annual and Semi Annual reports including the narratives and indicators d) Project quarterly financial information e) Field Visit reports available from the AOR f) Mid-term Assessment Report g) Any other reports pertaining to project performance that is available and required by the evaluation team. 60 Quantitative analyses (e.g. results against targets annually and over the life of the project by partner; trends in capacity building assessment scores and other methods of capacity measurement by partner; cost-benefit or return on investment analysis, as appropriate); Case studies of successful and unsuccessful local organizations; Key informant interviews and focus group discussions with a wide range of stakeholders including, but not limited to: a) USAID Zambia HIV/AIDS Multisectoral Office b) USAID Zambia Office of Financial Management c) LPCB implementing agencies (FHI 360 and MSI) d) LPCB Beneficiary organizations (local partner organizations and Capacity Leaders and Organization Development Facilitators) e) Community beneficiaries and leaders f) Other USAID and/or PEPFAR funded projects in Zambia, such as Support for HIV/AIDS Response in Zambia (SHARe II) g) National HIV/AIDS/STI/TB Council h) Other donors funding the local partner organizations and institution strengthening providers The evaluation team shall present to USAID/Zambia for review and approval a detailed data collection plan that details how and where data will be collected within as part of the work plan. C.5.3 Data Analysis Prior to the start of data collection, the evaluation team shall develop and present, for USAID review and approval, a data analysis plan that details how: (1) qualitative data such as key informant, stakeholder, and beneficiary interviews and/or focus group discussions will be transcribed and analyzed; (2) quantitative data will be analyzed and presented to determine trends over time, including dummy tables; and (3) the evaluation will weigh and integrate qualitative data from these sources with data from project capacity assessments, service delivery data, and project monitoring records to reach conclusions and recommendations. Analysis of qualitative and quantitative data should be used to generate a list of the most competent and promising local partner organizations and institution strengthening providers. Where needed, data will be disaggregated and analyzed by gender, type of organization, etc. C.5.4 Challenges Associated with the Required Evaluation There is a lack of a rigorous design and tools to assess the outcomes of the capacity building interventions. While there is some quantitative pre and post data on the capacity of the organizations supported by the project, such data is limited to scores that can be misleading due to the subjective nature of capacity building assessments. In the absence of good indicators and reliable methodologies that were originally set and applied by the project to assess organizational improvements, the evaluators will need to come up with creative ways to assess the capacity of organizations supported by the project which will require reconstructing the baseline to have a clearer picture of changes that have occurred within these organizations. Reconstructing the baseline may cause recall bias which may affect the quality of the data and evaluation findings. As part of the work plan, the contractor is required to propose a rigorous design and evaluation methodology despite all the potential data limitations in order to increase the rigor and credibility of the evaluation results. C.6 DELIVERABLES 1. Final Evaluation Design and Methodology: The contractor shall submit and obtain approval from USAID/Zambia for a detailed evaluation design prior to initiating any in country work. 2. Work plan: The contractor shall submit a detailed work plan aligned to the approved evaluation design within six days of arrival in the country. 3. Briefings: The Evaluation Team Leader shall brief the USAID Agreement Officer’s Representative (AOR) at the onset of the assignment, weekly during the course of the evaluation as schedule permits, and at the end of the assignment (before leaving the country). The evaluation team shall organize and provide entry, mid-term, and final briefings for USAID/Zambia staff, other USG agencies and staff, implementing partners, select local partners, and host government officials. 61 4. Interview Notes and List of Resource Documents: The Evaluation Team shall provide USAID/Zambia summaries of all key meetings, workshops, and discussions conducted during the course of the evaluation and copies of any relevant documents and reports gathered during the evaluation. 5. Summary Presentation of Findings to USAID/Zambia and Stakeholders: Two business days prior to departing Zambia, the evaluation team shall present initial findings to USAID/Zambia for review, comment and feedback. A PowerPoint presentation and handout (maximum of two pages) shall be prepared for the presentation. The team shall also present major findings of the evaluation to stakeholders. The team shall consider USAID/Zambia and stakeholder comments and revise the draft report as appropriate. 6. Evaluation Report: A draft evaluation report is due five business days after the field visit is completed. Within 10 business days of receiving USAID/Zambia’s feedback to the draft report, two hard copies and one electronic (MS Word) copy of the final evaluation report are due to USAID/Zambia. The evaluation report shall include the following: a. Executive Summary b. Background; c. Introduction; d. Methodology; e. Findings, including Lessons Learned; f. Conclusions; g. Recommendations; and h. Annexes, including: i. Scope of Work ii. Data collection tools iii. Key data sets, including interview transcripts iv. List of key informants v. Documents consulted The evaluation report shall meet the criteria for quality evaluation reports specified in Appendix I of the Evaluation Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf). If USAID/Zambia disagrees with any aspects of the report, the evaluation team is expected to include a section in the report describing the points of disagreement. C.7 EVALUATION TEAM COMPOSITION USAID/Zambia supports increased use of in-country experts and organizations for the implementation of evaluations. Therefore, inclusion of at least one local experienced evaluator is encouraged, but not required. The evaluation team should ideally consist of a team leader and at least three technical experts each contributing unique skills and expertise in project evaluation design and methods, organizational development and capacity building, skills building, and community-based HIV/AIDS programming. Consultants shall provide references for similar work conducted in the past 5 years and must not have any conflict of interest with the project’s Prime Partner, FHI360, MSI, or the LPCB local institution strengthening providers. It is encouraged that a representative from the National HIV/AIDS/STI/TB Council (NAC) be invited to participate to foster country ownership of the capacity building process. Evaluation Team Leader/Senior Evaluation Specialist The Evaluation Team Leader is responsible for the successful completion and documentation of the LPCB project final evaluation. The Team Leader will provide leadership for the team. S/he will work closely with the AOR and other staff as necessary to finalize the evaluation design, methodology, report outline, timeframe, the selection of site visits and meetings, the distribution of materials and other activities as identified. The Evaluation Team Leader will be responsible for the timely submission of all drafts and a quality final report to the Team Leader of the USAID/Zambia HIV/AIDS Multisectoral Office. The team leader must have the following profile: Education: An advanced degree (Ph.D., MPH, MA, MS, MBA or equivalent) in public health, epidemiology, demography, social sciences, organizational development, monitoring and evaluation or related fields. Work Experience: Minimum of ten years of progressively responsible experience with at least five years of evaluation experience with recognized organization(s) in the design, implementation, and evaluation of development programs, including experience working in developing countries. Experience working in Africa, and in particular Zambia, preferred. 62 Skills and Abilities: Demonstrated strong analytical, managerial, leadership and writing skills are critical for this assignment. The Evaluation Team Leader should be able to interact and communicate effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. The Evaluation Team Leader must be fluent in English and have proven abilities to communicate clearly, concisely, and effectively, both orally and in writing. Team Members The team members work under the guidance and direction of the Evaluation Team Leader and are responsible for contributing to the successful completion and documentation of the LPCB project final evaluation. To successfully complete the task, USAID/Zambia expects the candidates to have the following profile and skills mix: They candidates should have an advanced degree in a health or social science field with at least five years of experience in their field of expertise (e.g. project evaluation design and methods, NGO capacity building, skills building and organizational development, community-based HIV/AIDS service development and delivery). They must have experience in conducting evaluations or research studies and have excellent written and spoken English language skills. C.8 SCHEDULING, LOGISTICS AND SUPPORT The evaluation will be carried out over a period of approximately eight to nine weeks. Proposal must include a timeline based on the parameters described in the Deliverables section and above, which includes offeror’s suggested time for field work and writing the draft report The evaluation team shall submit a detailed timeline as part of the work plan to USAID/ Zambia. Extensive travel throughout Zambia is anticipated. The evaluation team shall not receive logistical support for travel, other than visa letters. Proposal budgets should include cost and brief description of transportation requirements. To facilitate field visits, USAID will provide introductions to key stakeholders, including: LPCB, local partner organizations, government counterparts and other stakeholders. USAID/Zambia personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. 63 ANNEX II: EVALUATION METHODS AND LIMITATIONS INTRODUCTION USAID/Zambia requested Social Impact, Inc. (SI) to conduct an end-of-project evaluation of the Local Partners Capacity Building (LPCB) Project. Through a cooperative agreement under the Capable Partners Leader with Associates Award, LPCB was awarded to the former Academy for Educational Development (now under FHI 360) in partnership with Management Systems International (MSI). The Project began on May 19, 2008 with a completion date of June 30, 2013 and a life-of-project cooperative agreement amount was $49,080,196. The purpose of LPCB is to improve the overall capacity of selected local NGOs, FBOs, and CBOs, collectively known as Partner Organizations (POs), to ensure efficient and effective expansion of their programs so that quality HIV/AIDS services in Zambia are delivered. LPCB has a life-of-project amount of $49,080,196. Ever since its onset,, LPCB has been a flagship program under USAID in Zambia to build the management, financial, technical, and M&E capacities of Zambian organizations. It is also directed at supporting selected intermediary support organizations and individuals to strengthen the capabilities and expertise of local organizations. LPCB was designed to contribute to the USAID/Zambia 2004-2010 strategic objective of Reduced Impact of HIV/AIDS through a Multisectoral Response, and associated intermediate results, as well as Zambia’s annual PEPFAR prevention, care and systems strengthening targets. The project’s capacity building models resonate USAID and PEPFAR’s emphasis on country ownership. PURPOSE AND USES OF THE EVALUATION The purpose of this evaluation is to assess program effectiveness and efficiency in order to inform future programming in strengthening local organizational capacity. For the purpose of this evaluation, the following definitions will apply: An activity is relevant if it is a required and necessary input for achieving an identified and clearly defined development outcome; An activity is effective if it produces the desired result, or development outcome it was designed to produce; An activity is efficient if it achieves its intended results within the allotted time and financial parameters; Sustainability is achieved when host country partners and beneficiaries demonstrate ownership of development processes, by taking on financial and managerial responsibility for maintaining and leveraging the project momentum and achievements beyond the life of the USAID project. The overall objectives of the evaluation are: 4. To assess the extent to which project objectives, targets, outputs and expected results were achieved and or exceeded (performance) in accordance with the LPCB Performance Monitoring and Evaluation Plan. 5. To assess the effectiveness and potential sustainability of project activities and capacity building model(s) and approaches on the institutional and technical capacity of local organizations and subsequent expansion of quality community-based HIV/AIDS services. Specifically, the evaluation will examine the success of the project in: 64 a. Increasing the institutional capacity and sustainability of local partner organizations to manage funds and programs efficiently; b. Improving the technical capacities of local partner organizations to implement and expand quality HIV/AIDS prevention, care and support activities and services effectively; and c. Establishing a cadre of experts and local intermediary organizations competent and committed to strengthening institutions and service providers 6. To inform USAID/Zambia of lessons learned and best practices for replicating and scaling up local capacity development models and identify promising and high performing local partners (replicability). The evaluation will also identify and document changes in LPCB’s model and activities since the Mid-Term Assessment (e.g. additional means for measuring organizational capacity, development and implementation of graduation plans and number of organizations that have graduated). It is anticipated that the evaluation will be useful to multiple audiences and stakeholders including inter alia: USAID/Zambia: To identify effective practices and areas for improvement and to inform the design and implementation of future interventions in building local organizational capacity in Zambia. To identify promising and competent local partner organizations among the POs and institution strengthening providers that could become viable direct local partners of USAID. USAID/Global Health Bureau, PEPFAR: To demonstrate the effectiveness and efficiency of engaging and strengthening the capacity of local organizations in expanding the delivery of quality HIV/AIDS services. USAID/Policy Planning and Learning Bureau: To suggest a model for evaluating capacity building programs. Government of the Republic of Zambia, and other donors: To demonstrate the effectiveness of strengthening local organizational capacity in expanding the delivery of HIV/AIDS services. LPCB, POs, and Institutional strengthening providers: To learn their strengths and weaknesses and adjust their technical approaches for future projects accordingly. USAID/Zambia will use findings from the evaluation to inform the design of future projects and publications on capacity building interventions for local organizations. The report will be disseminated widely among relevant stakeholders and project beneficiaries as well as submitted to the Development Exchange Clearing House (DEC). EVALUATION SCOPE AND FRAMEWORK Per USAID guidelines, a performance evaluation assesses the extent to which a program has achieved the targets set out at its inception. Due to the lack of valid control groups of local organizations, a non￾experimental performance evaluation design will be employed for this evaluation. The approach will utilize a systematic and comprehensive review of program outputs and outcomes on beneficiaries, using alternative methods to investigate the effect of LPCB activities on participating organizations. This evaluation is intended to measure quantitative and qualitative changes that have occurred in subject organizations, their staff, and broader communities. To the extent possible, the evaluation team will make before and after comparisons using organizations’ self-reported capacity scores and progress toward established targets; however, the team acknowledges the effect that subjective measures have on the internal validity of data. Therefore, triangulation of data from multiple sources will be used to corroborate self-reported data. 65 Quantitative M&E data maintained by POs and collected by LPCB will be used to track PO progress toward achieving PEPFAR and national HIV/AIDS indicator targets. HIV/AIDS outcome data available will be analyzed to extrapolate the contribution of LPCB on reducing the burden of HIV/AIDS in PO target areas. It is understood, however, that in the absence of appropriate data to conduct an impact evaluation, it will not be possible to attribute any measurable changes in HIV/AIDS prevalence or other individual PO project outcomes solely to LPCB inputs. The primary tools for collecting qualitative data to assess project organizational and managerial performance will be site visits and field observation coupled with key informant interviews and focus group discussions (FGDs) with PO management and service provider staff, as well as Capacity Leaders (CLs), Organizational Development Facilitators (ODFs), LPCB project staff, and USAID/Zambia personnel. The qualitative data collected at the different levels of LPCB implementation will contextualize quantitative analyses and help identify successful structural, managerial, or project design strategies worthy of replication. Thematic Analyses The Key Evaluation Questions (KQs) will be addressed via three methodological vantage points that incorporate the following thematic areas: Beneficiary Analysis of partner and intermediary organizations receiving support through the LPCB mechanism and the communities they serve using quantitative and qualitative performance measures, drawn from LPCB M&E performance monitoring data and key informant interviews, Organizational Capacity Assessment based on qualitative measures, including observed and reported changes in the technical, managerial, and overall organizational capacity of LPCB partner organizations; , Sustainability of observed and reported LPCB capacity building achievements through a comparative analysis of the LPCB capacity building model and partner organizations’ potential to maintain managerial and technical capacity changes beyond the life of the project. Key Evaluation Questions The evaluation will address a constellation of interrelated set of KQs: 6. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? 7. To what extent were the project design, implementation, and management effective and why? 8. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? 9. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? 10. What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? SOURCES OF DATA AND DATA COLLECTION ACTIVITIES To address each of these Key Evaluation Questions, the evaluation team will rely on a variety of data sources and data collection methods. The Data Collection and Analysis Matrix in Annex 1 is organized around each of the evaluation’s 5 Key Questions (KQs), and provides a description of data collection methods to be used. Annex 1 further describes the variety of data to be extracted and analyzed, the sources(s) of that data, and the types of analyses that will be undertaken to inform findings and conclusions. 66 KQ1 explores the extent to which LPCB achieved its stated objectives; thus, the question will be addressed using data from LPCB work plans, PMPs, quarterly progress reports, the Mid-term Assessment Report, other M&E data from the LPCB database, as well as complementary interviews with PO staff and CLs. To address the effectiveness of LPCB design, implementation, and management effectiveness asked in KQ2, the evaluation team will conduct interviews with key LPCB staff (FHI 360 and MSI), Institutional Strengthening Partners (ISPs), CLs, PO management staff, and key USAID/Zambia personnel. Here, data drawn from the LPCB M&E database will be used to analyze the relationship between support provided to POs and measures of PO capacity. The effectiveness of the LPCB model will be analyzed in relation to similar capacity building approaches. This analysis, coupled with data collected through qualitative and quantitative methods, will allow for the identification of priority areas for future capacity building interventions. KQ3 is concerned with the measurable changes in operational, management, and technical capacity induced by LPCB support; this will be addressed using PO self-reported capacity ratings, progress reports and other M&E data, as well as detailed interviews with PO staff. Particular emphasis will be placed on measuring organizational development relative to the level of PO existing capacity, as organizations of differing capacity levels participated in the project. POs at a higher level of organizational development will presumably have had different experiences than those with low levels of organizational development; therefore, POs of varying size and capacity will be selected for site visits and in-depth inquiry. KQ4 asks how LPCB has contributed, if at all, to increasing the delivery of quality HIV/AIDS services, which is best addressed using PO data reported on project and PEPFAR indicators, district-level HIV/AIDS service delivery data (if available), interviews with PO service delivery staff, and time/logistics permitting, interviews with community and partner focal points, and qualitative data from focus group discussions (FGDs) with CLs. Finally, KQ5 explores whether the capacity building efforts made by LPCB will be sustainable. To address sustainability, LPCB and individual PO financial data will be re-examined, in addition to in-depth interviews with LPCB project staff, CLs, and key stakeholders within POs. All data will be disaggregated by sex, where appropriate. Selection of Project Sites USAID has requested that the evaluation team sample 6 POs from each cohort under the LPCB project, for a total of 30 POs, with 30% selected from among organizations working in rural areas. In addition to the selected POs, the team will interview at three of the four Capacity Leader Organizations based in Lusaka and the Copperbelt, and two Organizational Development Facilitators (ODFs). The total sample size, therefore, will comprise 35 organizations, or approximately one third of all LPCB participants. Given the non-experimental nature of the evaluation’s design, it cannot be assumed that those organizations will be fully representative of the universe of project beneficiaries. However, the team will endeavor to include as representative as possible a cross section of the project’s universe of beneficiaries by employing a purposive sampling strategy based on characteristics of priority to the focus of this evaluation, such as: - PO Type: Non-governmental organizations (NGOs), faith-based organizations (FBOs), and community based organizations (CBOs) - Organization Focus and Mission, such as preventative HIV/AIDS services, community outreach; advocacy, home based care, skills training, etc. - Target Population, to ensure representation of POs targeting most at-risk populations (MARPs), orphans and vulnerable children (OVC), adolescents, and families - PO Characteristics such as organizational size, gender of executive - Program Longevity: Represent POs from each cohort representing each phase of project engagement (e.g. Capacity Assessment, Training, Funding, Graduation, etc.) 67 - Organization Performance, to ensure representation of both high- and low-performing POs - Geography: Represent, as time and logistics permit, POs in focal areas of LPCB implementation across urban and rural regions Based on the above selection criteria, a total of 35 Capacity Leader Organizations and Partner Organizations were selected for site visits. The complete list of selected organizations may be found in Annex 4. Qualitative Methods Key informants are the main data source for this mixed methods evaluation. Findings gleaned from key informant interviews will be triangulated with quantitative data to assess the project’s success in increasing technical and institutional capacity of local POs. The team will conduct interviews with a wide range of stakeholders including inter alia: USAID Zambia HIV/AIDS Multisectoral Office USAID Zambia Office of Financial Management LPCB implementing agencies (FHI 360 and MSI) LPCB Beneficiary organizations (local partner organizations and Capacity Leaders and Organization Development Facilitators) Community beneficiaries and leaders Other USAID and/or PEPFAR funded projects in Zambia, such as Support for HIV/AIDS Response in Zambia (SHARe II), STEPS-OVC and COH III National HIV/AIDS/STI/TB Council Other donors funding the local partner organizations and institution strengthening providers LPCB project partners’ management and technical staff will be interviewed to gauge perceptions of their organization’s capacity, what kind of tangible improvements the organization has experienced as a direct result of the project, and to what extent the project has improved the organization’s ability to serve its catchment population. They will also be asked to comment on project elements that have been the most beneficial, remaining constraints and perceived project weaknesses, and areas for improvement in future approaches to capacity development. A key focal area will assess the success of LPCB’s mandate to encourage and sustain local ownership of capacity development efforts in support of PEPFAR’s national HIV/AIDS goals and targets. Appreciative inquiry, through which informants are asked about the value the project has added to their workplace, community, and everyday life will be employed as deemed appropriate. Particular emphasis will be placed on selecting key informants that reflect an equal gender balance, per USAID Evaluation Policy guidelines. Ensuring that gender remains a key informant selection factor also aids the evaluation team in analyzing the effects of the project on women, women-owned organizations, and women in the community. Semi-structured interview guides tailored for each group of key informants will be developed, with the data recorded, translated as needed, and transcribed. The participatory and dynamic nature of focused group discussions is highly conducive to gleaning critically important information from beneficiaries’ experience with LPCB. Assuming conditions permit, the team may also reach out to secondary beneficiaries through focused group discussions. To mitigate recall bias and respondent subjectivity, qualitative data will be triangulated with available quantitative data and an effort will be made to frame questions commensurate with the context and culture those being interviewed. Annex 2 provides a template key informant interview guide with sample questions for various project stakeholders. 68 Qualitative Beneficiary Analysis and Capacity Assessment The goal of this type of analysis is to determine the perceived qualitative benefits accrued to CLs, POs, and their surrounding communities. Primary beneficiaries of LPCB assistance include CLs and POs – individuals and organizations receiving direct technical and financial assistance through the project model. Individuals accruing benefits of increased organizational capacity, such as individuals in communities receiving HIV/AIDS services from participating POs, can be considered secondary beneficiaries of LPCB. Qualitative analyses of both tiers of beneficiaries will be conducted with a view to identifying the extent to which LPCB assistance has contributed to improved outputs and outcomes at the organization and community levels. The team will make a concerted effort to examine the distribution of project benefits to organizations headed by women relative to those headed by men. Comprehensive analysis of findings will enable the evaluation team to ultimately identify which POs possess the capacity necessary to become direct recipients of USAID funding. The identification of common factors found to contribute to the success of high-performing POs will be particularly instructive for program implementers and stakeholders invested in perpetuating capacity development efforts throughout Zambia. Qualitative thematic areas will be triangulated with correlated quantitative data drawn from the LPCB Performance and Monitoring Evaluation Plan. For example, where POs are asked to describe the effects of LPCB assistance on their organizational capacity and service delivery, responses will be linked to PO capacity self-assessment scores and number of target population reached with HIV/AIDS services. Questions asking about PO capacity and strategies for increasing funding levels from non-LPCB sources will be triangulated with quantitative outcome data on the value of external grants awarded and the number of POs reporting an increase in funding sources over time. Similarly, qualitative data elicited on the perceived strengths and weaknesses of LPCB assistance will provide contextual evidence to quantitative measures such as the number of PO performance indicators above and below 75% of targets. Annex 2 provides a comprehensive data triangulation matrix stratified by each Key Evaluation Question. Site visits coupled with key informant interviews with CLs and PO staff will be conducted to cast light on the internal and external factors affecting the ability of organizations to achieve their stated objectives. A focused literature review of related research and surveys (Zambia Demographic and Health Survey, for example) will be carried out as needed to contextualize findings from interviews and identify community￾level benefits of LPCB. The principal tool for collecting data will be semi-structured interview guides based on the project’s Performance Monitoring and Evaluation Plan, tailored to each respondent group (e.g. CLs, PO management staff, PO service delivery staff, etc.). Questions posed of PO management staff will address areas such as the project’s effect on the organization’s operational capacity, while thematic areas explored with PO service delivery staff will focus on the effects of the project on the delivery of HIV/AIDS services to communities. Sample questions for POs include: What impact has the project had on your organization? How would you describe its impact on your everyday work? Your community? What has been the most useful component of this project to your organization’s capacity development? How has your organization’s ability to achieve its mission changed since participating in LPCB? How has training received through LPCB affected the way your organization manages or delivers services? Did you receive a commodity or service delivery grant? If so, how did the grant affect your organization’s ability to deliver quality HIV/AIDS services? How has the project affected your organization’s funding from sources other than LPCB? Why did your organization feel the need to develop its organizational capacity? What were your initial expectations? Have your expectations been met, and why or why not? 69 Do you feel the skills and knowledge you have gained from LPCB will continue to be useful after the project has ended? Why? What do you see as the future of your organization? What does your organization still need in order to maintain, improve, or expand services? Sample questions for CLs and ODFs include: What were your expectations of the project before participating? Were your expectations of the project and what you would gain from participation ultimately met? How have you used the skills you have gained as a result of this project? What has been your contribution to the project itself? Qualitative LPCB Model Analysis This analysis will focus primarily on the effectiveness of the LPCB model as a means of delivering useful and sustainable capacity development assistance to participating intermediary and partner organizations. Specifically, the management and organization of the project will be assessed against the project’s M&E indicators with a view to identifying strengths, weaknesses, and opportunities for improving the model of provision of capacity development assistance. A key area focal area of this analysis will be the sustainability of capacity development efforts beyond the life of the project. Further, the specific motivations prompting POs to participate in the project will be explored to identify the extent to which project structure met the needs and expectations of local organizations. Project staff and personnel from government agencies (MOH) will also be interviewed to explore how the LPCB project has helped mitigate the effects of understaffing in local organizations by building on existing capacity. To gather this evidence, the evaluation team will conduct in-depth interviews with key LPCB management staff, CLs, ODFs, and PO staff during site visits. Semi-structured interview guides will be tailored to each group of stakeholders. Findings will be triangulated with quantitative analyses of PO financial data to aid in the determination of whether LPCB participation helped POs to diversity and increase their funding sources and levels beyond LPCB grants. Analysis of the project’s capacity building model will yield critical information on best practices for the design and implementation of future capacity-related interventions, including program components worthy of replication and areas for improvement. Sample questions for POs include: Tell us about the technical assistance that CLs, ODFs and LPCB staff provide. What are some of the strengths and weaknesses of the support you’ve received? How frequently do you interact with your CL or ODF? How responsive is your CL/ODF to your organization’s needs? How can this communication be improved? Tell us about the Monitoring and Evaluation your organization conducts as part of LPCB. How effectively do LPCB’s M&E tools track/capture changes in your organizational capacity? How can these tools be strengthened? Describe some of the challenges you have faced in participating in LPCB. This can include challenges in data reporting, training logistics, management, etc. Tell us about your experience with the LPCB program structure. What are your impressions of how the activities were planned, managed, and what it was like to work with LPCB? In your opinion, how effective has this model been for developing your organization’s capacity? Describe some things that you might recommend in terms of improvements that can be made to the program if it were expanded or implemented elsewhere. Sample questions for project staff (technical, management, and executive) How efficient is the M&E reporting process? How often are reporting deadlines met? 70 How do you ensure that M&E data collected is useful/relevant to the project? How does project M&E relate to the national M&E framework? What is the role of the National AIDS Council (NAC) and how does NAC involvement ensure relevance of M&E to national HIV/AIDS targets? How has the project furthered the goals of national HIV/AIDS strategies? Quantitative Methods To the extent possible, the evaluation team will apply a quantitative analysis to answer KQs related to the performance of the project against targets. Data gathered over the life of the project as part of PO M&E, including capacity self-assessment scores, will be extracted and analyzed for trends and objective themes that reveal the extent to which PO capacity has been developed and sustained. Data to be requested by the evaluation team include: LPCB activities against work plans; LPCB indicator data against targets; LPCB PEPFAR impact and compliance indicators; PO assessment and institutional improvement data; PO participation and progress; PO grants, sub-agreement funding; and Grantee/sub-recipient progress against work plans, indicators and PEPFAR output, impact and compliance indicators.2 Other quantitative analyses will be conducted using PO financial data, which will attempt to draw relationships between levels of support provided and extent of organizational capacity built. Of particular importance to this evaluation is the level and diversification of financial input to POs from non-LPCB sources; therefore, where available, PO financial data will be analyzed to compare LPCB financial input to grants received from external sources. The quality of analysis from the above data will depend heavily on the validity of the data itself; the team will attempt to address this constraint by 1) explicitly stating limitations to generalizability of findings; and 2) triangulating quantitative measures with qualitative data. Recognizing that specific and valid PO financial input data may not be available, the evaluation team will explore PO financial status using qualitative inquiry. Data will be extracted from the LPCB M&E and National AIDS Council (NAC) databases, as well as POs and select interviews. Some data identified through the review of relevant literature and survey data may also be extracted to inform our quantitative analyses. Pending the availability and quality of data, tables containing key output results will be developed to determine trends across the life of the project and results against targets. Data will be disaggregated and presented by individual PO, LPCB cohort, and province. To the extent possible, data will also be disaggregated by sex (including comparisons between men- and women-led organizations). Tables 1-4 below provide illustrative data tables for outputs over the life of LPCB. 2 LPCB Associate Cooperative Agreement, May 2008 71 Table 1. Comprehensive Life-of-Project (LOP) Output Aggregation – Project Level LPCB OBJECTIVE OUTPUT INDICATOR LOP TARGET LOP ACTUA L 1. LPCB POs trained in core organizational functions and HIV/AIDS service delivery # of health care workers who successfully completed an in-service training program within the reporting period 1,380 # of improvement plans developed 100 2. Zambian CLs trained to provide capacity￾building support to smaller Zambian POs # of health care workers who successfully completed an in-service training program within the reporting period 70 Number of local organizations to which CLs/ODFs provide training or TA 100 3. Zambian professionals trained as organization development facilitators (ODF) # of health care workers who successfully completed an in-service training program within the reporting period 34 # of ODFs licensed to facilitate the IDF assessment with HIV/AIDS matrix 15 Establishment of a professional OD association in Zambia Yes 4. Grants provided to Zambian POs to strengthen their organizations and implement expanded HIV/AIDS services # of POs that received grants through LPCB during the reporting period 50% USD value of grants awarded during the reporting period TBD 5. Collaboration and networking promoted for Zambian HIV/AIDS organizations # of networking opportunities offered to stakeholders during the reporting period 30 Establishment of an LPCB networking website Website includes contributions from >50 POs # of organizations listed in LPCB database 500 Table 2. Comprehensive LOP Output Aggregation – PO Level PARTNER ORGANIZATION Client Satisfaction Score IDF (Capacity Self￾Asssessment) Score USD value of grants awarded from non-LPCB sources # target population reached with HIV prevention interventions Baseline End-of￾Project Baseline End-of￾Project Baseline End-of￾Project Baseline End-of￾Project Ex: Kafue Child Development Agency (KCDA) 72 Table 3. Comprehensive LOP Output Aggregation – Cohort and Provincial Level COHORT # POs meeting 75% service delivery targets # POs reporting client satisfaction increase # POs reporting increase in IDF Score # POs reporting increased grant awards from non￾LPCB sources # POs graduated from LPCB 1 2 3, cont. PROVINCE Central Copperbelt Lusaka, cont. Table 4. Comparison of Support Provided to Results COHORT # PO recipients of LCPB service delivery grants LCPB service delivery grant amount total Value of grants awarded from non-LCPB sources LCPB financial input % total funding # PO grant recipients that achieve 75% of service delivery targets 1 #/total USD USD % #/total 2 3, cont. PROVINCE Central Copperbelt Lusaka, cont. An additional quantitative measure that the evaluation team may introduce is an electronic or web-based mini-survey distributed to all 103 POs. Using Likert scales, survey questions will specifically address organizational capacity prior to LPCB support, perceptions of project efficacy, perceived strengths and weaknesses, and opportunities for sustainability of capacity. Though easy to develop and distribute, the utility of this type of quantitative tool is limited by the technological capacity of respondent organizations and subjectivity of responses. Due to the ex post nature of this type of collection, retrospective data is also subject to recall bias. Pending the availability of the above data, statistical analyses may be performed to identify correlations between LPCB inputs and key project outcomes, such as increased service delivery. The combination of other quantitative and qualitative data will allow the evaluation team to determine further correlational relationships; however, due to the ex post nature of the evaluation and anticipated lack of robust baseline data, causal inferences will not be able to be made. Data for these quantitative analyses will be derived from the LPCB M&E database, as well as individual PO databases (collected through site visits) and key LPCB project staff, such as M&E Specialists. As LPCB is responsible for collecting and generating summary reports of aggregate project achievements on a quarterly, semi-annual, and annual basis, LPCB project staff is likely to be able to make requested data available to the evaluation team. The availability of all data requested, 73 such as PO funding sources and amounts, is not guaranteed; therefore, the proposed quantitative component of this evaluation will depend heavily on the level and quality of data actually available. DATA ANALYSIS AND TRIANGULATION The Data Collection and Analysis Matrix in Annex 1 provides an overview of how different data and analysis processes are intended to lead to concrete findings, conclusions and recommendations that are responsive to the evaluation’s Key Questions. Interlinked methodological components of the LPCB performance evaluation will be carried out simultaneously, allowing for the continued triangulation of qualitative and quantitative findings throughout the data collection period. Triangulated findings will then form the basis for the synthesis of project results and formulation of cohesive, policy-relevant conclusions and recommendations. For example, measures used in quantitative beneficiary analysis will be linked to corresponding lines of qualitative inquiry in interviews with beneficiaries. For example, if PO service delivery gaps are observed in quantitative data through underperforming performance indicators, qualitative data from interviews with PO staff will be analyzed to reveal potential capacity constraints and other hindrances perceived to affect service delivery. Annex 2 provides a detailed matrix of how qualitative inquiry will be triangulated with quantitative data. QUALITY ASSURANCE In order to ensure that data of the highest quality is collected and analyzed, the evaluation team will first consult with LPCB and Mission staff to determine the extent to which available data is complete and likely to be accurate. The identification of potential weaknesses in available data at the onset of the evaluation will aid the team in focusing refining its data collection tools to ensure that data gaps are adequately addressed. The consistent triangulation of quantitative and qualitative data in the data analysis phase will ensure that findings are drawn from evidence of the highest possible quality. SI employs a three-stage QA process for all of its evaluations to ensure high quality, evidence-based results that are useful for program improvement, accountability, and learning purposes. Each stage of the evaluation is reviewed and vetted through checklists and direct feedback is given to the Team Leader and field team. Stage I: Work plan — The Senior Technical Advisor will review the feasibility and rigor of the proposed methodology and work plan and adequacy of the dissemination plan. Stage II: Draft Evaluation Report — Report structure and logical linkages among findings, analysis, conclusions, presentation of qualitative and quantitative data, and actionable recommendations will be assessed. Stage III: Final Report — A 40-point quality check of the executive summary, program, and methodology description; adequacy of findings analysis, conclusions, and final recommendations; full compliance with USAID evaluation policies; and report presentation, e.g. charts, graphs, and annexes will be conducted. REPORTING Following fieldwork, the evaluation team will prepare and deliver a presentation to USAID/Zambia consolidating data collected into formulation of preliminary findings, conclusions and recommendations. Based on feedback from the presentation, the team will draft a high-quality evaluation report consistent with the standards set forth in USAID’s Evaluation Policy. 74 ANNEX 1. Data Collection and Analysis Matrix Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 6. To what extent did the project achieve the planned objectives and results for capacity building as set out in the project agreement, approved PMPs and agreement amendments? Comparative/ Analytic Document review Data Abstraction LPCB M&E database LPCB Work Plans, progress reports N/A Compare observed and reported outputs and outcomes with indicator targets (indicators include PEPFAR as well as LPCB indicators Interviews to understand challenges in meeting targets and revisions to targets. Semi-structured interviews Key informants from beneficiary organizations and local partners Purposive 7. To what extent were the project design, implementation, and management effective and why? Comparative/ Analytic Document Review Data Abstraction Project financial statements LPCB M&E database N/A Analyze relationship of activity levels to output measures; to the extent possible, compare LPCB financial input to project results. Content analysis of interviews data to detect key themes related to implementation and management effectiveness Semi-structured Interviews Key informants from POs Purposive 75 Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 8. To what extent did the local organizations or individuals receiving support from LPCB experience measurable changes in their operational, management, and technical capacity, including funding levels and sources beyond the LPCB grants? Comparative/ Analytic Document review Data Abstraction Statistical Analysis (as applicable) Data abstraction from LPCB M&E database PO internal capacity assessment score data N/A (Targeted M&E information on LPCB partners) Analyze relationship of support received to measures of capacity Use qualitative methods followed by content analysis of interviews to uncover themes in tangible capacity development of LPCB beneficiaries. Semi-structured interviews Key informants among LPCB beneficiaries Purposive 9. What has been the project’s contribution to increasing delivery of quality HIV/AIDS services? To what extent can the increase in service delivery of quality HIV/AIDS services, if any, be linked to increased organizational capacity? Comparative/ Analytic Document review Statistical Analysis (as applicable) LPCB M&E database Results against PEPFAR indicators N/A (Targeted M&E information on LPCB partners) Use quantitative analysis to determine the effect of LPCB on service delivery indicators and outputs at the PO level Content analysis of key informant and beneficiary interview data. Triangulation with observations from field visits to project sites, quantitative and qualitative beneficiary analyses. Semi-structured interviews Stakeholders, key informants, beneficiaries Purposive 76 Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 10. What are the prospects for the sustainability of the capacity building results for the local partner organizations and the institution strengthening providers? Analytic Semi-structured interviews Stakeholders, key informants, management and service delivery staff of beneficiary organizations Purposive Content analysis of key informant interviews to discover likelihood of sustainability of project results. 77 ANNEX 2. Data Triangulation Matrix KEY EVALUATION QUESTION ILLUSTRATIVE LINES OF QUALITATIVE INQUIRY LINK TO QUANTITATIVE ANALYSIS Achievement of planned objectives/results In what ways has PO organizational capacity increased? # HCWs successfully completing in￾service training (change over time) How would PO rate its organizational capacity? IDF Score change over time How has LPCB affected CL/ODF ability to serve communities? # of improvement plans developed How do improvement plans (based on IDF process) lead to PO capacity development? Comparison of # improvement plans developed with IDF Score over time What factors facilitated/inhibited POs from developing organizational capacity? % of PO performance targets met Effectiveness of LPCB design, implementation, and management What are the strengths and weaknesses of LPCB management structure? % of PO performance targets met How responsive are CLs/ODFs to POs’ needs? # organizations provided technical assistance by CLs/ODFs # hours of technical assistance provided to POs How effectively do M&E tools track/capture changes in organizational capacity? Comparison of IDF score with # performance indicators above 75% target What challenges have POs faced data reporting and management of technical assistance? # POs and CLs that submitted data to NAC each quarter What do POs/CLs/ODFs perceive as most beneficial aspect of LPCB? % of PO performance targets met Measurable changes in beneficiary capacity and funding levels/sources How has LPCB influenced PO’s ability to seek funding from sources other than LPCB? # POs reporting increase in grant proposal submissions over time 78 How has PO organizational capacity changed since LPCB participation? % of PO performance targets met How capable are POs of becoming direct recipients of USAID funding? # POs graduated or near graduation from LPCB LPCB’s contribution to increased HIV/AIDS service quality and provision How has technical assistance improved the way POs manage or deliver services? # target population reached with HIV/AIDS services How do POs perceive the effect of assistance on community? PO client satisfaction change over time How has quality of PO service delivery improved as a result of technical assistance? PO client satisfaction change over time Sustainability of capacity development beyond life of project How confident are POs that they will graduate from LPCB? USD value of grants awarded to POs/CLs How will assistance POs/CLs received continue to be useful after LPCB has ended? # POs/CLs reporting increase in grants received from non-LPCB sources over time Where do POs anticipate future funding will be sourced from? # POs/CLs reporting increase in number of funding sources over time How will CLs/ODFs continue to develop capacity of local organizations? # trainings held for POs and CLs # ODFs licensed to facilitate IDF assessment # individuals trained as ODFs ANNEX 3. Template: Structured Key Informant Interviews STRUCTURED KEY INFORMANT INTERVIEW LCPB End-of-Project Evaluation Interviewee Name: Organization: Location: Cohort: Category of respondent: PO Management Staff E-mail contact: Date: Interviewer: Describe briefly how your organization became involved with LPCB. Why did your organization feel the need to develop its organizational capacity? What were your initial expectations? Have your expectations been met, and why or why not? What impact has the project had on your organization? How would you describe its impact on your everyday work? Your community? How has the project affected your organization’s ability to achieve its objectives? How has training received through LPCB affected the way your organization manages or delivers services? How has the project affected the delivery of HIV/AIDS services (quality and quantity)? What does your organization still need in order to provide quality HIV/AIDS services? Tell us about the technical assistance that CLs/ODFs provide. What are some of the strengths and weaknesses of the support you’ve received? What is the relationship like between you and your CL/ODF? Has this relationship met your expectations? How frequently do you interact with your CL/ODF? How responsive is your CL/ODF to your organization’s needs? How can this communication be improved? Tell us about the Monitoring and Evaluation your organization conducts as part of LPCB. How effectively do LPCB’s M&E tools track/capture changes in your organizational capacity? How can these tools be strengthened? How do you utilize the data you collect? How would you suggest improving the reporting process? Describe some of the challenges you have faced in participating in LPCB. This can include challenges in data reporting, training logistics, management, etc. Tell us about your experience with the LPCB program model. What are your impressions of how the activities were planned, managed, and what it was like to work with LPCB? In your opinion, how effective has this model been for developing your organization’s capacity? How has the project affected your organization’s funding from sources other than LPCB? (e.g. number of funding sources, amount of overall funding, etc.) Do you feel the skills and knowledge you have gained from LPCB will continue to be useful after the project has ended? Why? Please be specific. Evaluator’ s comments INTERNA L USE ONLY 80 What do you see as the future of your organization? What does your organization still need in order to maintain, improve, or expand services? Describe what you would do differently to improve the project if it were expanded or implemented elsewhere. ANNEX 4. Sources Selected for Interviews USAID/Zambia Cynthia Bowa, LPCB AOR Ngaitila Phiri, Alternate AOR Ky Lam, HIV/AIDS Multi-sectoral Team Leader FHI360/MSI Chad Rathner, ZPI COP Cara Endyke-Doran, LPCB COP Francis Johnston, LPCB DCOP Namute Malama, LPCB Capacity Building Manager Mike Welsh, FHI 360 Country Director USAID-Funded Projects SHARe II STEPS-OVC COH III NAC Representatives Capacity Leaders Southern African AIDS Trust (SAT) Zambia Interfaith Networking Group on HIV and AIDS (ZINGO) Zambia Health Education Communications Trust (ZHECT) Copperbelt Health Education Project (CHEP) Organization Development Facilitators Alvin Nchemba Chalwe M Nyirenda Christian Chileshe Daisy Kambandu Daniel Lyatumba Felix Banda Gideon Bulwani Ignatius Kayawe Kennedy Musonda Mwenda M Mumbuna Tenso Kalala 82 Partner Organizations AATAZ - Lusaka Kafue Gospel Singers Community Project (KGSCP) - Kafue Youth Development Organisation (YDO) – Choma Chilanga Youth Awake – Lusaka New Masala Theatre Group – Ndola Dambwa Christian Care Centre – Livingstone Youth Cultural Promotions Association (YOCUPA) – Lusaka Bridge of Hope Foundation – Chipata Chisomo Home Based Care – Chipata Katete District Women Development Association – Katete Grassroot Soccer – Lusaka Mboole Rural Development Initiative (MRDI) – Choma Zambia Voluntary Community Development Association (ZAVCODA) – Livingstone Ndola Catholic Diocese – Ndola Roan Youth Development – Luanshya Zambia Centre for Communication Programmes (ZCCP) – Lusaka Restless Development – Kabwe Jubilee Centre – Ndola Catholic Medical Mission Board (CMMB) – Lusaka HODI – Lusaka Judith Chikonde Foundation – Mufulira Family Health Trust – Lusaka Community Youth Mobilisation (CYM) – Kabwe Treatment Advocacy and Literacy Campaign (TALC) – Lusaka HIV and AIDS Prevention Network (HAPN) – Mufulira Afya Mzuri – Lusaka Harvest Help Zambia – Siavonga Zingo South – Livingstone Eastern Province Women Development Association – Chipata NZP+ Petauke – Petauke Chadiza Women Development Association – Chadiza Lifeline Zambia – Lusaka Kwenuha Women’s Association - Livingstone SEPO Centre - Livingstone 83 ANNEX 5. LPCB Project Sites by Province Central Province Cohort 3: 2, Kapiri Mposhi (1), Kabwe (1) Cohort 4: 4, Kapiri Mposhi (2), Kabwe (2) Cohort 5: 1, Kabwe Copperbelt Province Cohort 1: 1, Ndola Cohort 2: 4, Ndola (1), Kitwe (3) Cohort 3: 4, Ndola (2), Kitwe (1), Luanshya (1) Cohort 5: 2, Mufulira (2) Eastern Province Cohort 1: 4, Chipata (3), Lundazi (1) Cohort 2: 4, Chipata (2), Katete (3), Lundazi (1) Cohort 3: 1, Chipata Cohort 5: 4, Chipata, Lundazi, Chadiza, Petauke Luapula Province Cohort 1: 3, Kawambwa (1), Mansa (2) Cohort 2, 4, Mansa (2), Nchelenge (2) Cohort 3: 2, Mansa (2) Cohort 5: 1, Mansa Lusaka Province Cohort 1: 5, Kafue (3), Lusaka (2) Cohort 2: 6, Lusaka (3), Kafue (2), Chongwe (1) Cohort 3: 3, Lusaka (3) Cohort 4: 7, Lusaka (Lusaka) Cohort 5: 1, Lusaka Muchinga Province Cohort 4: 2, Mpika (2) Cohort 5: 4, Mpika (3), Chinsali (1) Northern Province Cohort 4: 3, Kasama (3) Northwestern Province Cohort 3: 1, Solwezi Cohort 4: 4, Mwinilunga (2), Kasepa (1), Solwezi (1) Cohort 5: 4, Kabompo (3), Zambezi (1) Southern Province Cohort 1: 5, Livingstone (4), Choma (1) Cohort 2: 4, Livingstone (2), Choma (2) Cohort 3, 1, Choma Cohort 5, 3, Livingstone, Siavonga, Mazabuka Western Province Cohort 3: 4, Mongu (4) Cohort 5: 4, Senanga (2), Mongu (1), Kaoma (1) ANNEX 6. Work Plan ACTIVITIES JUNE JULY AUGUST 1 2 3 4 1 2 3 4 1 2 3 4 In -brief Meeting with USAID/Zambia (3 June ) Methodology and Work Plan Finalization (9 June) Site Visits to CLs and POs (Team 1) Site Visits to CLs and POs (Team 2) Meetings with USAID and LPCB staff Data Analysis Out -brief Presentation (8 July ) Draft Report Preparation and Submission (15 July) USAID/Zambia Report Review Final Report Preparation and Submission (9 August) 85 Week 1 In-brief meeting with USAID/Zambia (3 June) Meetings with LPCB staff Methodology and Work Plan finalization (9 June) Week 2 Site visits in Copperbelt Province (Team 1) Site visits in Southern Province (Team 2) Interviews with Lusaka-based CLs Week 3 Site visits in Eastern Province (Teams 1 and 2) Focus Group with ODFs (21 June) Interviews with NAC Representatives Week 4 Site visits in Lusaka and Central Province (Teams 1 and 2) Interviews with USAID/Zambia, USAID-funded project staff Week 5 Data analysis and draft report preparation Week 6 Summary of findings presentation to USAID/Zambia (8 July) Week 7 Draft report submission (15 July) Weeks 8 and 9 USAID/Zambia review of draft report Week 10 Final report preparation Week 11 Final report submission (9 August) 86 ANNEX III: DATA COLLECTION INSTRUMENTS STRUCTURED KEY INFORMANT INTERVIEW- Partner Organizations LCPB End-of-Project Evaluation Date Organization Location Cohort Interviewee Name(s) Position Title(s) E-mail contact Interviewer 1 Describe briefly how your organization became involved with LPCB. Why did your organization feel the need to develop its organizational capacity? What were your initial expectations? Have your expectations been met, and why or why not? KQ 1 2 What impact has the project had on your organization? How would you describe its impact on your everyday work? Your community? KQ 3 3 How has the project affected your organization’s ability to achieve its objectives? How has training received through LPCB affected the way your organization manages or delivers services? KQ 3/4 4 How has the project affected the delivery of HIV/AIDS services (quality and quantity)? What does your organization still need in order to provide quality HIV/AIDS services? KQ 4 5 Tell us about the technical assistance that CLs/ODFs provide. What are some of the strengths and weaknesses of the support you’ve received? KQ 2 6 What is the relationship like between you and your CL/ODF? Has this relationship met your expectations? How frequently do you interact with your CL/ODF? How responsive is your CL/ODF to your organization’s needs? How can this communication be KQ 2 improved? 7 Tell us about the Monitoring and Evaluation your organization conducts as part of LPCB. How effectively do LPCB’s M&E tools track/capture changes in your organizational capacity? How can these tools be strengthened? How do you utilize the data you collect? How would you suggest improving the reporting process? KQ 2/3/4 8 Describe some of the challenges you have faced in participating in LPCB. This can include challenges in data reporting, training KQ 2 logistics, management, etc. 9 Tell us about your experience with the LPCB program model. What are your impressions of how the activities were planned, managed, and what it was like to work with LPCB? In your opinion, how effective has this model been for developing your organization’s capacity? KQ 2 10 How has the project affected your organization’s funding from sources other than LPCB? (e.g. number of funding sources, amount of overall funding, etc.) KQ 3 11 Do you feel the skills and knowledge you have gained from LPCB will continue to be useful after the project has ended? Why? Please be specific. 87 KQ 5 12 What do you see as the future of your organization? What does your organization still need in order to maintain, improve, or expand services? KQ 5 13 Describe what you would do differently to improve the project if it were expanded or implemented elsewhere. KQ 2 14 Is there anything else you would like us to know about your experience with the project? 88 LPCB Project Staff Interview Guide: Please give us a brief overview of the LPCB Program; its design and tools used. How is the project model design conducive to attaining the project’s objectives? Describe the evolution of Program design, implementation and management over the life of the Program. What specific challenges presented themselves that required changes to the initial design, implementation and management? How do LPCB executive management staff ensure that project targets are met? Describe the level of coordination between LPCB management staff and CLs/ODFs. Tell us a little about the process organizations undergo to apply for participation in LPCB. What criteria are used to screen participation? How were these criteria determined? What is anticipated for the POs not yet graduated, but still working on capacity strengthening? How will their capacity development be supported? What is the status of Cohort 5 at this time of Program phase-out? What do you consider to be the most critical indicators this project is tracking? Tell us a bit about data quality assurance. How is data recorded, tracked, and checked for validity? How is this M&E data ultimately used? Please be specific. What has been the impact of LPCB on the delivery of HIV/AIDS services in terms of quantity and quality? How are these impacts measured or tracked? Do you see a need for a follow-up program? If you were to design the follow-up program to LPCB, what lessons have you learned and what would you build into the follow-up program? What would you do differently? What component of this program holds the greatest potential for sustainability? Why? In your opinion, what has been the best thing about this project? What do you see as LPCB’s lasting legacy? 89 ODF Interview Guide: How did you get involved with LPCB? What motivated you to become an ODF? What were your expectations of this role and the project before you started? Were those expectations met? What have been your contributions to LPCB? What have you gained from LPCB? What were some of the challenges you faced personally? What have been some of your successes? What are your impressions of how LPCB was structured? Managed? Organized? Now that LPCB is over, how will capacity building efforts be sustained? What do you see as the future of capacity building projects in the HIV sector? If there were an LPCB 2, what would you do differently and why? USAID/Zambia Interview Guide: What expectations did you have for the project’s performance? Over the past 5 years, how has the project performed compared to your expectations? For a project centrally focused on building capacity, how does LPCB align with the objectives of PEPFAR? In your view, how effective has LPCB been at achieving the objectives of PEPFAR? What do you see as the link between organizational development and HIV service delivery? In the design phase of LPCB, how soon did you envision being able to see the effects of increased capacity on service delivery? There are capacity building components of the other projects in the Mission’s HIV Multisectoral Portfolio (COH III, SHARe II, and STEPS OVC). Could you give a brief overview of those capacity building elements and how they are incorporated into the projects themselves? How have these projects (LPCB included) interfaced to achieve the goal of capacity building? What is envisioned for future capacity building interventions in the HIV sector? To what extent is capacity building likely to be the central focus of future HIV-related programming? 90 ANNEX IV: SOURCES OF INFORMATION Documents Reviewed Title 2008 - 2012 Annual Reports 2009 – 2013 Semi-annual Reports LPCB Organizational Structure LPCB Performance Monitoring & Evaluation Plan LPCB Indicator Narrative LPCB M&E Filing Index LPCB Work Plan Addendum 2010-2011 LPCB Work Plan 2010-2011 LPCB Work Plan 2011-2012 LPCB FY 12 Success Stories Success Stories for ZPRs LPCB Midterm Assessment FHI 360 Final Report LPCB Institutional Development Framework Matrix PO Final Reports and Grants Data Key Informants Organization Name Title Email USAID/Zambia Ryan Washburn Deputy Mission Director rwashburn@usaid.gov Debi Mosel Program Officer dmorsel@usaid.gov Patricia Sitimela Senior M & E Specialist pstimela@usaid.gov Stella Mutale Deputy Program Officer snmutale@usaid.gov Chris Foley PDO cfoley@usaid.gov Arlene Phiri HIV Prevention Specialist aphiri@usaid.gov Ky Lam HIV/AIDS Multisection T/L klameu@usaid.gov Cynthia Bowa HIV/AIDS Multisection Deputy T/L cbowa@usaid.gov Emma Sitambuli Prog Dev Specialist/Gender Adviser esitambuli@usaid.gov Ayana Angulo Contracting officer aangulo@usaid.gov Martin Mikus OVC Advisor mmikus@usaid.gov Elizabeth Chisala Procurement Assistant echisala@usaid.gov SAT Zambia Caroline Magani Asst Program Officer (LBCB) carolinelombe@yahoo.com Martin Silukena Program Officer at LPBC project martin@satzam.org.zm Julius Kapambwe Regional Program Officer PCB Julius@satzam.org.zm Zoonadi Ngwenya Country Director ngwenya@satzam.org.zm Dambwa Christian Care Centre Mr. Kasongo Programme Manager ZAVCODA Chushi Kasongo Programme Manager zavcoda1998@gmail.com Bishop Smart Kobela Executive Director Joseph Mwale Finance Manager Society For Family Health Kelly Young Contract Finance Treatment Advocacy and Literacy Campaign Felix Mwanza Executive Director felixtalc@iconnect.zm Lusaka - Lifeline Zambia Florence Chileshe Nkhuwa Executive Director lifeline@zamnet.zm Afya Mzuri Lizzy Chanda Director of Programmes lizzyc@afyamzuri.org.zm 91 AATAZ - Lusaka Joseph Matafwali Accountant joematafwali@gmail.com Community Based For TB/HIV/AIDS Nason Banda Executive Director cbto2002@yahoo.com ZINGO Jeff Yusuf Ayami Executive Secretary admin.zingo@gmail.com ZHECT Chilufya Mwaba Phiri Director chilufyam@zhect.org.zm Godwin Banda Program Manager godwinb@zhect.org.zm Chadiza District Women’s Association Margaret Banda Administration Assistant mwendapat@yahoo.com CODEP Aubrey Banda Field Officer codechip@yahoo.com Canicious Chikambwe Programmes Katete- CINDI Children in Distress Ezekiel Sakala Coordinator ezekielsakalapeace2@yahoo.com Starford Lukopa Accountant lukopastar@hotmail.com Henry Banda Program Officer henryb774@gmail.com Katete- District Women’s Dev. Assoc Martrida Banda Program Officer matridabanda@yahoo.com Solomon Banda M & E kingsolomon73@live.co.za Viness Phiri Program Coordinator pvainess@yahoo.com NZP+ Petauke Columbia Changa District Coordinator sgroup@yahoo.com G. Zulu Program Officer Brown Mwanza Program Officer - Livehoods Samson Tembo Finance and Administration Bridge of Hope Foundation Ernest kabulansando Chief Executive Officer kabulansando@gmail.com Fwilane Banda Coordinator janerybanda@yahoo.com Chipata Eastern Dev Association Helen Banda Executive Secretary epwda@gmail.com Fadales Lungu Administration Assistant Sara Makarainen Volunteer Jesus Cares Ministries Pastor Godfrida Msumali Director jesuscares@microlink.zm Girl Guide Association Evelyn Mooba Coordinator info@guideszambia.com Afya Mzuri Victoria Mwiya Murgatroyd Executive Director enquiries@afyamzuri.zm Flame Community Based Org Dorothy Charity Brolund Executive Director flamecharity@yahoo.com Mwendalubi kauseni Outreach Officer Winnie Bwalya Mwila Project Manager Faith Musonda Kasonde Secretary faithkasonde@yahoo.com Expanded Church Response Precious Soko Asst Program Manager precious.soko@ecrtrust.org Yvonne Pande Technical Coordinator yvonne.pande@ecrtrust.org Harvest Help Zambezi Alexander Kasenzi Executive Director harvesthelp@zamtel.zm National Aids Council Dr. Clement Chela Director General cchela@nacsec.org.zm Douglas Hampande Civil Society Coordinator dhampande@gmail.com 92 RICAP Michael Lungu Program Manager michaellungu50@yahoo.com Samuel Tundu Director ricapkafue@gmail.com Community Youth Mobilization Abisheck Musonda Executive Director cymlstone@zamtel.zm Restless Development Chola Kunda M & L Coordinator chola@restlessdevelopment.org Ruth Asimwe M & L Manager ruth@restlessdevelopment.org Zambia Center for Communications Program Mandy Dube Programmes Coordinator Kafue Child Development Cacious Miyanda Program Coordinator David Muriya M&E Officer John Siantanga Grants Accountant Kafue Gospel Singers Charles Chishimba Project Coordinator kgscp@yahoo.com Family Health Trust Humphrey Menda Finance Manager fht@zamtel.zm ZPI Chad Rathner Chief of Party crathner@fhi360.org Ndola Catholic Diocese Sister Mombwe Head of Diocesan Health Services timsakala@gmail.com HAPN Marjorie Tuba Makumba Director hapnproject@yahoo.com MRDI Jonsen Hamachimba Executive Director Michael Hachibize M&E Jonny Hamachimba Admin Assisstant RYDP Partner Siabutuba Executive Director YDO Christine Phiri Program Manager CHEP Ronnie Jere Sibu Malambo CTYA Program Manager Program Coordinator Judith Chikonde New Masala Theatre Group Andrew Chisanga Executive Director newmasala@yahoo.com CBTO IODF Alvin Nchemba Organization Development Facilitator Daniel Lyatumba Chairman, IODF Ignatius Kayawe Organization Development Facilitator 93 ANNEX V: LPCB INDICATORS REPORTED PEPFAR Indicators Reported: P7.1.D – Number of people living with HIV/AIDS (PLHIV) reached with a minimum package of prevention with PLHIV (PwP) interventions P8.1.D – Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet minimum standards required P8.2.D – Number of the targeted population reached with individual and/or small group level HIV prevention interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence and/or meet the minimum standards required (subset of P8.1.D) P8.3.D – Number of MARPs reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required P11.1.D – Number of individuals who received testing and counseling (T&C) services for HIV and received their test results H2.3.D – Number of new health care workers who successfully completed an in-service training program within the reporting period INDICATOR RESULTS # Target population reached with individual and/or small group level HIV prevention interventions that are based on evidence and/or meet the minimum standards required 220,890 people Level of increase in PO client satisfaction - # POs/CLs with increase in annual IDF score 80% POs # PO grant recipients that achieve at least 75% of stated objectives 77% POs # POs/CLs that acquire increased grant or in-kind resources from non-LPCB sources 80% POs # of POs/CLs that submitted data to NAC in previous quarter 88% POs # health care workers who successfully complete in-service training program 8,937 people # of local organizations provided TA by CL/ODF 107 POs # ODFs who successfully complete in-service training (# individuals trained as ODFs) 13 people # ODFs licensed to facilitate IDF 13 people # POs that received LPCB grants 90% POs # networking opportunities offered to partners 62 POs 94 Establishment of LPCB networking website Yes Establishment of professional OD association Yes 95 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings Action for Positive Change (APC) - Chipata Eastern Chipata 1 Completion - SD 36 306,128,758 306128758 13 3 16 Action for Social Development Foundation (ASDF) - Mansa Luapula Mansa 1 Participation -15 113,532,000 113532000 13 3 16 Adolescent Reproductive Health Advocates (ARHA) - Mongu Western Mongu 3 Graduation -17 223,777,638 223777638 13 3 16 Afya Mzuri - Lusaka Lusaka Lusaka 4 Graduation -21 992,505,944 992505944 11 2 13 Anti-AIDS Teachers Association of Zambia (AATAZ) - Lusaka Lusaka Lusaka 1 Graduation 24 199,669,660 199669660 13 3 16 Bethesda- Zambia – Kitwe Copperbelt Kitwe 3 Graduation 24,892,397 121,563,596 146455993 13 3 16 Bridge of Hope Foundation - Chipata Eastern Chipata 2 Graduation 70 356,302,200 356302200 12 3 15 Bumi Bwesu Youth Centre - Nchelenge Luapula Nchelenge 2 Completion - CG -8 19,668,864 n/a 19668864 12 3 15 Caritas Mansa - Mansa Luapula Mansa 3 Completion - CG -12 15,952,190 n/a 15952190 10 0 10 Catholic Diocese of Solwezi - Solwezi Northwestern Solwezi 3 Completion - SD 6 159,257,600 159257600 13 3 16 Catholic Medical Mission Board (CMMB) - Lusaka Lusaka Lusaka 3 Drop n/a n/a 0 11 1 12 Central Action on HIV and AIDS (CAHA) - Kabwe Central Kabwe 4 Drop 225,483,700 225483700 12 3 15 Chadiza Women Development Association – Chadiza Eastern Chadiza 5 Completion - CG 80 24,856,621 126,379,620 151236241 16 4 20 Chilanga Youth Awake - Lusaka Lusaka Lusaka 1 Completion - SD 20 305,534,959 305534959 13 3 16 Children In Distress (CINDI) Chimasuko - Katete Eastern Katete 2 Completion - CG -10 23,748,999 n/a 23748999 11 2 13 Children In Distress Project (CINDI) – Kitwe Copperbelt Kitwe 2 Graduation 4 311,473,370 195,661,273 507134643 12 3 15 Chisomo Home Based Care - Chipata Eastern Chipata 2 Graduation 68 24906903 221,547,380 246454283 11 2 13 Community Based TB/HIV/AIDS Lusaka Lusaka 2 Graduation 22 325,220,942 124,627,540 449848482 12 3 15 96 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings Organization (CBTO) - Lusaka Community for Human Development - Lusaka Lusaka Lusaka 4 Graduation 6 24861724 125,246,950 150108674 11 2 13 Community Oriented Development Programme (CODEP) - Chipata Eastern Chipata 1 Completion - CG 22 23,258,310 23258310 13 3 16 Community Youth Mobilisation (CYM) - Kabwe Central Kabwe 4 Completion - CG 40 24995862 24995862 11 2 13 Contact Trust Youth Association (CTYA) - Livingstone Southern Livingstone 1 Graduation 32 319,787,760 189788100 509575860 13 3 16 Dambwa Christian Care Centre - Livingstone Southern Livingstone 1 Completion - CG 28 22,628,836 22628836 13 3 16 Development Organisation of People Empowerment – Mpika Muchinga Mpika 5 Completion - CG 1 24,595,621 24595621 15 3 18 Dorcamo Community HIV/AIDS Prevention and Care – Kapiri Mposhi Central Kapiri Mposhi 4 Participation 49 24608103 24608103 11 3 14 Eastern Province Women Development Association – Chipata Eastern Chipata 5 Completion - SD 32 24,828,121 121,250,007 146078128 15 5 20 Expanded Church Response (ECR) - Lusaka Lusaka Lusaka 4 Graduation 621,182,935 621182935 11 2 13 Family Health Trust - Lusaka Lusaka Lusaka 4 Graduation 109,745,950 109745950 12 3 15 Flame Community Based Organization - Lusaka Lusaka Lusaka 2 Participation -25 23428448 n/a 23428448 12 3 15 Fleet of Hope – Kabwe Central Kabwe 5 Drop 24,960,621 24960621 16 3 19 Foundation for Wildlife and Habitat Conservation – Mpika Muchinga Mpika 5 Drop 125,487,136 125487136 13 0 13 Girl Guide Association of Zambia - Lusaka Lusaka Lusaka 4 Participation -22 n/a n/a 0 11 2 13 Global AIDS Africa Foundation - Chongwe Lusaka Chongwe 2 Participation -23 24857069 n/a 24857069 12 3 15 Grassroot Soccer - Lusaka Lusaka Lusaka 2 Drop 16 125,120,941 125120941 10 2 12 97 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings Groups Focused Consultants - Mansa Luapula Mansa 3 Graduation 192,556,575 192556575 12 2 14 Harvest Help Zambia - Siavonga Southern Siavonga 5 Graduation 24725000 126,036,700 150761700 17 4 21 HIV and AIDS Prevention Network (HAPN) – Mufulira Copperbelt Mufulira 4 Drop 24,859,070 24859070 16 3 19 HODI - Lusaka Lusaka Lusaka 3 Drop -6 1,649,760,825 1649760825 12 2 14 Jesus Cares Ministries (JCM) - Lusaka Lusaka Lusaka 4 Participation 431,958,150 431958150 11 2 13 Jubilee Centre - Ndola Copperbelt Ndola 3 Completion - SD 127,119,740 127119740 12 2 14 Judith Chikonde Foundation - Mufulira Copperbelt Mufulira 4 Graduation -1 24,701,000 121,619,892 146320892 17 4 21 Kabompo AIDS Program – Kabompo Northwestern Kabompo 5 Completion - CG 1 17,360,622 17360622 16 3 19 Kafue Child Development Agency (KCDA) - Kafue Lusaka Kafue 1 Graduation 4 317,014,311 185783345 502797656 13 3 16 Kafue Gospel Singers Community Project (KGSCP) - Kafue Lusaka Kafue 1 Completion - SD 16 24437500 124,931,440 149368940 11 3 14 Kaoma Youth Alive – Kaoma Western Kaoma 5 Participation 24,334,070 24334070 17 4 21 Kasama Christian Community Care – Kasama Northern Kasama 4 Graduation 200,339,635 200339635 13 3 16 Kasama Young Media – Kasama Northern Kasama 4 Completion - CG -7 24429310 24429310 12 2 14 Katete District Women Development Association - Katete Eastern Katete 2 Graduation -13 21092786 220,379,000 241471786 12 3 15 Kubalusa Community Based Organisation - Kasempa Northwestern Kasempa 4 Participation 15 119,844,765 119844765 10 1 11 Kwenuha Women Association - Livingstone Southern Livingstone 1 Graduation 25 25,829,943 178,667,623 204497566 12 3 15 Lifeline Zambia – Lusaka Lusaka Lusaka 5 Participation 22 n/a n/a 0 15 2 17 Luanshya Support Group – Luanshya Copperbelt Luanshya 2 Completion - CG 20 24865524 24865524 11 2 13 98 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings Luapula Families in Distress (LUFAID) - Mansa Luapula Mansa 1 Completion - SD 13 328,714,500 328714500 13 3 16 Luapula Foundation - Mansa Luapula Mansa 2 Graduation -8 1,226,085,443 1226085443 8 0 8 Maluba Home Based Care – Chinsali Muchinga Chinsali 5 Completion - CG 24,469,121 24469121 16 3 19 Mansa District Women Development Association - Mansa Luapula Mansa 3 Participation 45 n/a n/a 0 13 0 13 Manyinga Community Development Trust – Kabompo Northwestern Kabompo 5 Participation 9 n/a n/a 0 13 0 13 Maveve OVC and HBC – Kabompo Northwestern Kabompo 5 Drop 8 24,886,570 24886570 16 3 19 Mboole Rural Development Initiative (MRDI) - Choma Southern Choma 2 Completion - CG -4 23,974,137 23974137 12 3 15 Mpika Diocese - Mpika Muchinga Mpika 4 Participation 27 n/a n/a 0 12 2 14 Mthuzi Development Foundation (MDF) - Chipata Eastern Chipata 1 Drop 31 160,557,371 160557371 12 3 15 Mwinilunga NZP+ - Mwinilunga Northwestern Mwinilunga 4 Completion - CG -14 24525689 24525689 12 3 15 NASCENTS - Mpika Muchinga Mpika 5 Drop 22,350,000 22350000 16 3 19 Luapula Nchelenge 2 Graduation 24 320,649,939 320649939 12 3 15 Ndola Catholic Diocese - Ndola Copperbelt Ndola 2 Drop -9 125, 547,863 0 12 3 15 Network Of Zambian People Living with HIV and AIDS - Mansa Luapula Mansa 2 Completion - SD 22 249,207,834 249207834 9 0 9 Network of Zambian People Living with HIV/AIDS – Choma Southern Choma 2 Completion - CG 4 24850011 24850011 12 3 15 New Masala Theatre Group - Ndola Copperbelt Ndola 1 Completion - CG 61 25,726,250 25726250 11 3 14 Northern Health Education Programme (NOHEP) - Kasama Northern Kasama 4 Participation 22947500 22947500 12 2 14 99 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings NZP+ Lundazi – Lundazi Eastern Lundazi 5 Participation 1 22,010,621 22010621 17 4 21 NZP+ Mongu - Mongu Western Mongu 5 Drop 33 n/a n/a 0 15 2 17 NZP+ Petauke – Petauke Eastern Petauke 5 Completion - CG 2 17,812,282 17812282 15 3 18 People’s Participation Services (PPS) - Mongu Western Mongu 3 Completion - CG 18 24,634,310 24634310 13 3 16 Pride Community Health Club – Kafue Lusaka Kafue 2 Completion - CG 69 22,470,793 22470793 12 3 15 Prisons Fellowship - Ndola Copperbelt Ndola 3 Graduation 16 118,490,288 118490288 13 3 16 Program for Vulnerable Children and Women (PVCW) - Mpika Muchinga Mpika 4 Completion - CG 38 24716724 24716724 13 3 16 Ray of Hope - Livingstone Southern Livingstone 2 Participation -2 n/a n/a 0 12 3 15 Restless Development - Kabwe Central Kabwe 3 Completion - CG 5 15,951,724 15951724 12 2 14 Rise Community Aid Program (RICAP) - Kafue Lusaka Kafue 2 Graduation 23 n/a 198,117,591 198117591 12 3 15 Roan Youth Development - Luanshya Copperbelt Luanshya 3 Graduation 15 22,840,968 121,911,936 144752904 12 2 14 SEPO Centre – Livingstone Southern Livingstone 1 Participation 41 n/a n/a 0 12 3 15 Sesha Life Savers – Senanga Western Senanga 5 Completion - CG 24,489,070 24489070 15 2 17 Simalelo AIDS Peer Education Program – Mazabuka Southern Mazabuka 5 Participation n/a n/a 0 13 2 15 Solwezi Youth Alive - Solwezi Northwestern Solwezi 4 Completion - SD 13 183,722,153 183722153 11 2 13 Tasintha Programme – Kapiri Mposhi Central Kapiri Mposhi 4 Completion - CG 23 24984482 24984482 11 2 13 Thandizani Community Based HIV/AIDS Prevention & Care - Lundazi Eastern Lundazi 1 Completion - SD 6 324,982,119 324982119 11 2 13 Treatment Advocacy and Literacy Campaign (TALC) - Lusaka Lusaka Lusaka 4 Participation 8 478,263,766 478263766 12 3 15 Tulipamo AIDS Support – Kapiri Mposhi Central Kapiri Mposhi 3 Graduation 6 212,032,050 212032050 13 3 16 100 Partner Organization (PO) Province Town Cohort Final Graduation Category % Diff IDF Score CG Value SD Value IG Value Total $ Received # Trainings # Elective Total # Trainings Twafwane Community Christian Centre - Kitwe Copperbelt Kitwe 2 Completion - CG -7 24937068 24937068 12 3 15 World Hope International Zambia (WHIZ) - Choma Southern Choma 3 Drop n/a n/a 0 12 2 14 Youth and Child Care Foundation (YCCF) - Mongu Western Mongu 3 Completion - CG 23,409,087 n/a 23409087 12 2 14 Youth Cultural Promotions Association (YOCUPA) - Lusaka Lusaka Lusaka 1 Drop 8 123,803,000 123803000 9 1 10 Youth Development Association (YDA) - Kawambwa Luapula Kawambwa 1 Completion - CG 33 22,608,616 22608616 12 3 15 Youth Development Organisation (YDO) – Choma Southern Choma 1 Graduation 42 358,376,620 197171431 555548051 12 3 15 YWCA - Mongu Western Mongu 3 Completion - CG 2 21,254,400 21254400 13 3 16 YWCA Senanga Branch – Senanga Western Senanga 5 Drop 5 23917283 23917283 16 3 19 Zambezi Development Trust – Zambezi Northwestern Zambezi 5 Drop 9 n/a n/a 0 16 3 19 Zambia Centre for Communication Programmes (ZCCP) - Lusaka Lusaka Lusaka 3 Graduation 12 1,684,711,212 1684711212 12 2 14 Zambia Voluntary Community Development Association (ZAVCODA) - Livingstone Southern Livingstone 2 Participation 3 24545603 24545603 12 3 15 Zingo South – Livingstone Southern Livingstone 5 Graduation 18 22,149,448 123,319,100 145468548 16 3 19 101 ANNEX VI: EVALUATION TEAM BIOS TOOMEY LINDSEY – Team Leader Health Systems Strengthening; Program Performance, Service Delivery Evaluation Expert Lindsey Toomey’s African health systems expertise is targeted to the design, implementation, management, impact evaluation, and quality improvement of developing healthcare delivery system interventions; models, tools and processes. She has led service delivery performance evaluations in South Africa of primary health care delivery systems and hospitals; working directly with the South African Department of Health. Lindsey has extensive experience in program performance and service delivery quality review within the health sector; to generate improved outcomes. Her health care delivery focus is on HIV/AIDS, TB, nutrition, maternal child health, family planning/reproductive health, and malaria. Lindsey is an expert in primary health care and hospital organisational capacity building, institutionalised and sustainable leadership, management, governance, and policy environment related to public health. She has the interpersonal skills to motivate and direct team functions and judgment regarding interventions required to achieve success. Resident in South Africa from 1993 through 2000, Lindsey was appointed by the ANC government to serve on the combined government/NGO body that demarcated the Eastern Cape health districts in 1994,. Working closely with civil society, government, and private and public health providers, Lindsey designed and implemented South Africa’s model district health delivery system in the Eastern Cape Province which was adopted in the Eastern Cape Province and subsequently replicated nationally. She authored the USAID Monograph on Functional Integration of District level PHC services. Lindsey worked with the Eastern Cape Minister of Health to establish the first ANC Dept. of Health. More recently, she undertook an impact evaluation of the USAID funded Integrated Primary Health Care Project (IPHC) in South Africa, designed to improve access and utilisation of child and reproductive health, nutrition, and HIV/AIDS services through integrated PHC service provision and health systems strengthening at the district, sub-district and facility levels. She currently serves on the Board of Lwala Community Alliance in Kenya, focused on maternal-child health in the primary health care setting and inpatient hospital maternity services. SABREEN ALIKHAN - Research Specialist At Social Impact, Ms. Alikhan manages a portfolio of short and long-term contracts, including performance evaluations for USAID, as well as impact evaluations for the Millennium Challenge Corporation. She provides managerial and technical guidance to evaluation teams, performs data analysis and report preparation, and manages client interface. Proficient in quantitative and qualitative evaluation methodologies, Ms. Alikhan has experience employing mixed-methods techniques to conduct performance evaluations for USAID projects in health and nutrition. Previously, Ms. Alikhan designed and directed a performance management plan and mixed-methods evaluation of mental health service delivery among urban, low-income youth and served as a research assistant for a pilot health project evaluating the usefulness and feasibility of offering complimentary HIV screening in urban hospital settings. In this capacity, Ms. Alikhan conducted semi-structured interviews with project participants and provided econometric and epidemiologic analysis on survey and clinical datasets. She also previously designed and conducted a longitudinal study in India to evaluate the impact of a women’s economic empowerment cooperative, formulating recommendations to improve microfinance programs. Trained in applied qualitative and quantitate research methods, Ms. Alikhan has completed graduate-level coursework in biostatistics, study design, quantitative methods and data analysis. Ms. Alikhan holds a Master of Global Health with a concentration in evaluation. 102 SULA NAKANYIKA-MAHONEY - Capacity Building Specialist Sula Nakanyika-Mahoney is an experienced program administrator with over 12 years of experience at a senior level in the coordination and implementation of Developmental, Gender and HIV/AIDS programs. Her experience in the Zambian private sector has enabled her to provide strategic direction and leadership for the various organizations she has worked for in designing a management structure, work plans, budgets and monitoring and evaluation processes. She has worked on numerous HIV/AIDS related projects in Zambia and brings strong local knowledge to the team. Most recently Ms. Nakanyika￾Mahoney served as a Gender Development specialist on an assessment of an HIV/AIDS project for the COMESA organization where she was in charge of evaluating the success of project goals. She also served as an HIV/AIDS specialist for Intrahealth International in their Zambia office. Ms. Sula Nakaniya￾Mahoney holds an MBA from the University of Glamorgan, is certified in Standard HIV/AIDS Peer Education, and is fluent in English, Nyanja, and Bemba. MOSES MBAWO – HIV/AIDS Specialist Moses Mbawo is a skilled program manager and trainer with over 12 years of experience working with donor-funded projects in the HIV/AIDS sector in Zambia. He has provided training to hundreds of individuals in the areas of HIV awareness and prevention, HIV in the commercial agribusiness sector, and training of trainers in HIV challenges in business administration. Mr. Mbawo has extensive experience working with USAID-funded Zambian HIV/AIDS initiatives, having served as an administrator for SHARe II, Market Access Trade and Enabling Policies (MATEP), and BizAIDS. Most recently, he served as Project Coordinator for the LEAD Program, wherein he led project monitoring and evaluation, coordinated HIV/AIDS activities involving the private sector, and trained Small Medium Entrepreneurs (SMEs) in business management systems and cross-cutting themes. As part of his work as Lead Consultant for the Zambia Chamber of Small & Medium Business Associations (ZCSMBA), Mr. Mbawo also conducted an evaluation on export readiness among women entrepreneurs in COMESA countries, and conducted extensive HIV/AIDS training and material development. Mr. Mbawo is a graduate of Zambia Insurance Business College and London University, and is fluent in English, Nyanja, and Bemba. MARYAM HASSAN - Research Assistant Maryam Hassan provides research, administrative and logistical support for Social Impact's range of evaluation training courses and evaluative work in Washington, DC and abroad. She also provides project management backstopping for both performance and impact evaluations based in Zambia, Jordan, and Guinea, among others. Before joining Social Impact, Maryam worked in the Joint Constitution Unit on Analytical Review of Somalia’s draft constitution at the United Nations Development Programme (UNDP) Somalia office in Nairobi, Kenya. During this time, she assisted in developing work plans, liaising with donors and partners, compiling research, and writing reports. She has a diverse range of research experience in international development, with particular emphasis on capacity building and good governance in Somalia. Maryam holds a Bachelor of Arts degree in International Relations and Middle Eastern Studies from the College of William and Mary, and is currently an MSc candidate in Development Management at the London School of Economics. She is fluent in English and Somali and is proficient in written and spoken Arabic. 103 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523