April 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mary O’Grady, Pat Sullivan, Sindiswe Nxumalo-Hleta, with support from Mary Pat Selvaggio of Khulisa Management Services (Pty) Ltd. EVALUATION Mid-term Evaluation of the Rapid and Effective Action Combating HIV & AIDS (REACH) Project in Swaziland Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Evaluation Team Ms. Mary O’Grady, Team Leader, Khulisa Consultant Ms. Pat Sullivan, Team Member, Khulisa Consultant Ms. Sindiswe Nxumalo-Hleta, Team Member, Khulisa Consultant Evaluation Manager Mary Pat Selvaggio, Khulisa Director Evaluation COR Charles Mandivenyi, USAID COR Cover Photo: Courtesy of Cabrini Ministries, St. Philip’s Mission, Mhlathuze, Swaziland. Credit: Richard Dalle Molle. MID-TERM EVALUATION OF THE RAPID AND EFFECTIVE ACTION COMBATING HIV & AIDS (REACH) PROJECT IN SWAZILAND: BUILDING CAPACITY OF LOCAL CIVIL SOCIETY ORGANIZATIONS WORKING IN THE HIV AND AIDS RESPONSE April 2014 Contract: AID-674-I-12-00002, Order No. AID-674-TO-14-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | i MAP OF SWAZILAND Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | ii TABLE OF CONTENTS Acknowledgements .................................................................................................................. v Acronyms ................................................................................................................................. vi EXECUTIVE SUMMARY......................................................................................................... 1 EVALUATION PURPOSE & EVALUATION QUESTIONS ............................................... 8 Evaluation Purpose .................................................................................................................................................................... 8 Evaluation Questions ................................................................................................................................................................ 8 PROJECT BACKGROUND ...................................................................................................10 EVALUATION METHODS & LIMITATIONS ....................................................................13 Sampling Framework ...............................................................................................................................................................14 Methodology .............................................................................................................................................................................15 Data Collection ................................................................................................................................................................16 Data Analysis .....................................................................................................................................................................17 Response Rates.........................................................................................................................................................................18 Limitations .................................................................................................................................................................................18 FINDINGS...............................................................................................................................20 Overview of PACT’s Capacity-Building Approach ..........................................................................................................20 Question 1. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? ..........................................................................................................................23 Findings ...............................................................................................................................................................................23 Conclusions .......................................................................................................................................................................33 Recommendations ...........................................................................................................................................................33 Question 2. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? .........................................................................................35 Findings ...............................................................................................................................................................................35 Conclusions .......................................................................................................................................................................42 Recommendations ...........................................................................................................................................................42 Question 3. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? ...................................................................................................43 Findings ...............................................................................................................................................................................43 Conclusions .......................................................................................................................................................................47 Recommendations ...........................................................................................................................................................47 Question 4. How does different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? .................................................48 Findings on geographic coverage and quality and Dosage .....................................................................................48 Findings on sub-partners Engagement and Alignment with national strategies and systems, and referrals to government services: .................................................................................................................................................51 Conclusions .......................................................................................................................................................................54 Recommendations ...........................................................................................................................................................55 Question 5. What approaches for delivering sustainable community services for OVC are working well? ..56 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | iii Overview ............................................................................................................................................................................56 Findings ...............................................................................................................................................................................58 Conclusions .......................................................................................................................................................................62 Recommendations ...........................................................................................................................................................63 Question 6. To what extent has PACT strengthened efforts to improve the OVC response beyond sub￾partner service delivery through activities related to developing of standards and other national processes? ......................................................................................................................................................................................................64 Findings ...............................................................................................................................................................................64 Conclusions .......................................................................................................................................................................66 Recommendations ...........................................................................................................................................................67 Question 7. What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? ............................................................................................................................68 Findings ...............................................................................................................................................................................68 Conclusions .......................................................................................................................................................................71 Question 8. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? ..................72 Findings ...............................................................................................................................................................................72 Conclusions .......................................................................................................................................................................74 Recommendations ........................................................................................................................................................... 74 OVERALL CONCLUSIONS ................................................................................................. 76 OVERALL RECOMMENDATIONS......................................................................................76 Annex I: Evaluation Statement of Work ............................................................................78 Annex II: Evaluation Methods and Limitations ..................................................................87 Annex III: Data Collection Instruments ..............................................................................97 Key Informant Interview Guide............................................................................................................................................97 Technical Capacity Assessment Tool (Technical CAT) ................................................................................................. 98 On line survey for all organizations ..................................................................................................................................104 Focus Group Discussion Tools for OVC projects ........................................................................................................106 Sub-Grantee Focus Group Questions ..............................................................................................................................110 Annex IV: Sources of Information .....................................................................................112 Documents Reviewed ...........................................................................................................................................................112 People Interviewed ................................................................................................................................................................113 Annex VI: Disclosure of Conflicts of Interest ...................................................................116 LIST OF FIGURES Figure 1. Beneficiaries reached by PACT sub-partners by HIV program (2011-2013) .............................................11 Figure 2. Conceptual Framework for Evaluation of Capacity Building ..........................................................................14 Figure 3. Comparison of Average OPI Scores for all sub-partners excluding new partners FY2011-2013 .........21 Figure 4. Overall OCAT results for PACT sub-partners 2011-2013 .............................................................................23 Figure 5. Governance and Leadership OCAT results for PACT sub-partners 2011-2013 ......................................24 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | iv Figure 6. MERL assessment results for PACT sub-partners 2011-2013 (except for new sub-partners in 2013) .......................................................................................................................................................................................27 Figure 7. Overall MCAT assessment results for PACT sub-partners 2011-2013 (except for new sub-partners in 2013) ........................................................................................................................................................................27 Figure 8. Accounting Procedures MCAT assessment results for PACT sub-partners 2011-2013 ........................27 Figure 9. Effectiveness of PACT’s technical HIV/AIDS capacity building of sub-partners .........................................37 Figure 10. Overall technical assessment results 8 sub-partners 2011-2014 ................................................................. 38 Figure 11. PREVENTION Technical Assessment results 3 partners 2011-2014 and 1 in 2014 ..............................40 Figure 12. TREATMENT and CARE technical assessment results 2 partners 2011-2014 ........................................40 Figure 13. OVC technical capacity assessment results 4 partners 2011-2014 ............................................................41 Figure 14. Increased range of sub-partners’ high-quality services based on PACT’s capacity building .................43 Figure 15. Expansion of beneficiary numbers based on PACT’s assistance..................................................................44 Figure 16. Appropriateness of Targeting based on PACT’s assistance..........................................................................45 Figure 17. OVC targeting and reach by goods and services according to OVC, parents/guardians, and community caregivers ..............................................................................................................................................47 Figure 18. Increased frequency of beneficiary support based on PACT’s assistance.................................................51 Figure 19. The effects of PACT’s assistance to sub-grantees in working with government ....................................53 Figure 20. Service effects of sub-grantee’s closer work with government...................................................................54 Figure 21. Response to the introduction of the projects to communities ...................................................................58 LIST OF TABLES Table 1. Sub-partner Organizations Funded by PACT and CANGO ............................................................................12 Table 2. PACT’s Capacity Development Stage Rating .......................................................................................................13 Table 3. Criteria for selecting the sample of partners to be interviewed.....................................................................14 Table 4. Sampled Organizations’ Length of experience with REACH project ............................................................15 Table 5. Overall Response Rates in the Evaluation .............................................................................................................18 Table 6. Response Rates in the Anonymous Online Survey.............................................................................................18 Table 7. Selected OD gaps in sub-partners’ ISPs.................................................................................................................24 Table 8. Organizational Capacity Building delivered by PACT 2012-2013 ...................................................................29 Table 9. Organizational Capacity-building Workshops – FY2011-2013 ........................................................................30 Table 10. Online survey respondents’ opinions of PACT’s organizational capacity building (N=22)....................32 Table 11. Technical Capacity Building Workshops conducted by PACT (FY2011-FY2013) ...................................35 Table 12. Umbrella Care Targets, Actuals, and Results for 5 sub-partners 2011-2013 ............................................45 Table 13. FY2013 targets and results for interventions implemented by 13 PACT sub-partners .........................46 Table 14. Sampled sub-partners’ range of services, regional coverage, reach and funding ......................................48 Table 15. OVC Reached by Cabrini and SCSWD (2012-2013)......................................................................................50 Table 16. Sampled sub-partner involvement in NSF 2014 to 2018 ................................................................................52 Table 17. Program Elements by sub-partners working with OVC (PACT funding only) .........................................56 Table 18. Community-level Respondents Average Satisfaction Scores for OVC services .......................................58 Table 19. OVC Standards pilot study key results ...............................................................................................................65 Table 20. CANGO OD Roadmap – Strengths and Weaknesses as rated by CANGO ...........................................68 Table 21. PACT sub-partner capacity-building workshops FY2011-2013 relevant to Global Fund capacity requirements ..............................................................................................................................................................73 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | v ACKNOWLEDGEMENTS The evaluation team wishes to thank everyone in Swaziland who accommodated our needs, in particular the many individuals who went out of their way amid busy schedules to provide various types of support. The team would like to thank all the staff members at various Government of Swaziland ministries, departments, and programs who met with us, including the Swaziland National AIDS Programme (SNAP) at the Ministry of Health, the Ministry of Education & Training, the National Children’s Coordination Unit (NCCU) and the Department of Social Welfare at the Deputy Prime Minister’s Office, and the National Emergency Council on HIV/AIDS (NERCHA), as well as UNICEF. The team also wishes to thank the USAID/Swaziland staff who supported this evaluation, especially Silke Felton, Natalie Kruse-Levy, Wendy Benzerga, Grace Masuku, and Patrick Kunene. Special thanks go to the PACT staff who supported this evaluation and all the sampled sub-partners that participated in the evaluation, including Bantwana Initiative, Cabrini Ministries, The Coordinating Assembly of Non-Governmental Organizations (CANGO), Khulisa Umntfwana, Lutheran Development Service (LDS), Save the Children Swaziland (SCSWD), Swaziland Business Coalition on Health and AIDS (SWABCHA), Swaziland National Network of People Living with HIV/AIDS (SWANNEPHA), and Voice of the Church (VOC). The team also wishes to especially thank all the individuals who participated in the focus group discussions conducted in the ten communities in Swaziland. This report is dedicated to the people of Swaziland. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | vi ACRONYMS AIDS Acquired Immune Deficiency Syndrome AMICALL Alliance of Mayors’ Initiative for Community Action on AIDS ANC Antenatal Care APR Annual Progress Report ART Antiretroviral Therapy ARV Antiretroviral BCC Behavior Change Communication CANGO Coordinating Assembly of Non-governmental Organizations CBO Community-based Organization CEO Chief Executive Officer CHW Community Health Worker DPM Deputy Prime Minister’s Office DSW Department of Social Welfare ECCD Early Child Care and Development ED Executive Director FBO Faith-based Organization FGD Focus Group Discussion FLAS Family Life Association of Swaziland FP Family Planning FY Fiscal Year GMU Grants Management Unit of NERCHA GSH Good Shepherd Hospital HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus HR Human Resources ICAP International Center for AIDS Care and Treatment Programs (Columbia University) IEC Information, Education and Communication ISP Institutional Strengthening Plan KAP Knowledge, Attitudes, and Practices KII Key Informant Interview LDS Lutheran Development Service M&E Monitoring and Evaluation MARP Most-at-Risk Population MDR-TB Multi-drug Resistant TB MER Monitoring, Evaluation and Reporting MERL Monitoring, Evaluation, Reporting & Learning MMC Medical Male Circumcision MNCH Maternal, Newborn and Child Health Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | vii MOET Ministry of Education & Training MOH Ministry of Health NATICC Nhlango AIDS Training Information and Counseling Centre NCCU National Children’s Coordination Unit NCP Neighborhood Care Point NERCHA National Emergency Response Council on HIV/AIDS NFM New Funding Model of the Global Fund NGO Non-governmental Organization NSF National Strategic Framework OCA Organizational Capacity Assessment OCAT Organizational Capacity Assessment Tool OD Organizational Development OVC Orphans and Vulnerable Children PEP Post-exposure Prophylaxis for HIV PEPFAR President’s Emergency Plan for AIDS Relief PHC Primary Health Care PMP Performance Monitoring Plan PMTCT Prevention of Mother-to-Child Transmission of HIV PNC Prenatal Consultation/Care PR Principal Recipient of Global Fund resources PSS Psycho-social Support RFA Request for Application RH Reproductive Health RHM Rural Health Motivator SBCC Social and Behavior Change Communication SCSWD Save the Children Swaziland SNAP Swaziland National AIDS Programme SOW Statement of Work SSR Super Sub-Recipient of the Global Fund to Fight AIDS, Tuberculosis and Malaria STI Sexually Transmitted Infection SWABCHA Swaziland Business Coalition on Health and AIDS SWAGAA Swaziland Action Group Against Abuse SWANNEPHA Swaziland National Network of People Living with HIV and AIDS TA Technical Assistance TB Tuberculosis TNS TechnoServe Swaziland UGM Umbrella Grants Mechanism UNICEF United Nations Children’s Fund US United States USAID United States Agency for International Development USG United States Government Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | viii VOC Voice of the Church WASH Water/Sanitation/Hygiene WFP World Food Programme WHO World Health Organization WORTH Women’s Empowerment Program Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The primary objectives of this mid-term performance evaluation of PACT’s Community Reach Project in Swaziland were to:  assess the quality of project implementation;  assess progress that has been made by the project toward achieving its set goal, objectives, expected outputs and/or outcomes;  determine which approaches and activities are working well and why, and  indicate areas that may require mid-term corrections. The 8 evaluation questions to be addressed were: 1. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? 2. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? 3. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? 4. How does different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? 5. What approaches for delivering sustainable community services for OVC are working well? 6. To what extent has PACT strengthened efforts to improve the OVC response beyond sub￾partner service delivery through activities related to developing of standards and other national processes? 7. What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? 8. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? PROJECT BACKGROUND Community REACH 2 in Swaziland is a five-year project (March 2010 to March 2015) managed by PACT and funded by USAID/Swaziland. The REACH project was designed to facilitate the efficient flow of grant funds and to deliver targeted technical assistance and capacity building services to organizations contributing to the HIV/AIDS response. The overall goal of REACH is to reduce the Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 2 impact of HIV/AIDS and improve health care for the people of Swaziland by developing a strengthened, coordinated civil society response to HIV/AIDS. In order to achieve this goal, the project is expected to achieve the following results: Result 1: To strengthen 1-2 local NGOs in umbrella grant management thereby enabling them to manage and sub-award funding from PEPFAR, Global Fund and other major donors in Swaziland. Result 2: To provide PEPFAR funding and minimal technical assistance to 3-5 mature NGOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 3: To provide PEPFAR funding and a full package of technical assistance to 10-15 nascent NGOs/CBOs/FBOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 4: Provide an as yet to be determined number of Global Fund recipients with training, mentorship and linkages to appropriate technical assistance. PACT operates as an Umbrella Grant Mechanism (UGM) with a focus on strengthening the organizational effectiveness of NGOs to improve their capacity in project management, financial accountability, and monitoring and evaluation for results-oriented programming and organizational sustainability. PACT’s mandate is to build the capacity of organizations in a number of critical areas including grants and finance management, technical and program management, organizational development, monitoring, evaluation, reporting and learning. Toward this end, PACT currently supports 14 partners delivering HIV services in various technical areas which include sexual prevention, HIV testing and counseling, care and treatment, TB/HIV, impact mitigation, and gender. Some of PACT’s sub-partners operate in all four of Swaziland’s administrative regions, while others have a more limited reach. PACT also supports the Coordinating Assembly of NGOs (CANGO), in developing its capacity to serve as a local UGM for civil society organizations in Swaziland for other donors, with a specific focus on the Global Fund under the National Emergency Response Coordinating Agency (NERCHA). NERCHA lists CANGO as a ‘super-sub-recipient’ tasked to serve as an umbrella body for civil society organizations. Five PACT sub-partners shifted over the last 15 months to be managed directly by CANGO with close mentoring and supervision from PACT. In FY2013 PACT supported CANGO to undertake a competitive selection process for three new sub-partners, which started to receive funding in August 2013. EVALUATION DESIGN, METHODS AND LIMITATIONS This Performance Evaluation utilized a non-experimental design that excluded a rigorously-defined counterfactual or comparison group. However, the evaluation team incorporated before-after comparisons to determine changes in project activities, outputs and outcomes over time. There are 17 organizations (i.e. sub-partners) affiliated with PACT’s REACH project, including 4 for which funding ended in September 2013. A subset of eight (8) organizations were selected for in￾depth interviews plus CANGO for a total of 9 organizations to be selected for in-depth data collection The team employed a mixed methods approach (collection of both qualitative and quantitative data) to collect both primary and secondary data and to ensure that the eight evaluation questions were answered comprehensively. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 3 The various methods employed in this Performance Evaluation included: a) Document and data review; b) Key informant interviews (KIIs) with PACT managers and staff; a selection of the nine sampled sub-partner managers and staff and relevant stakeholders; c) Focus Group Discussions (FGDs) with male and female beneficiaries, parents and caregivers affiliated to OVC sub-partner programs in 10 sites; and FGDs with the selection of the eight sampled sub-partner managers and staff plus CANGO; d) Technical Capacity Assessments with the eight relevant sub-partners; e) Anonymous online survey of staff and managers for all 17 sub-partners. The mix of qualitative and quantitative data which was analyzed using methods appropriate to each. Some of the techniques used to analyze the data included:  Theme analysis, which involved: o Viewing the data several times as a whole o Identifying patterns and themes o Reorganizing the data (e.g., coding the data according to the themes identified).  Triangulation: o Cross-checking the data in order to increase the confidence in the findings o Use of multiple data sources (PACT and sub-partner managers and staff as well as beneficiaries) o Use of multiple data collection methods (document review, Technical Capacity Assessments, KIIs, anonymous online survey and focus group discussion).  Descriptive Statistics including graphs: o Trend Analyses o Frequencies o Percentages o Composite Scores Potential bias was possible from all respondents, including selection bias from the OVC beneficiaries, caregivers and parents/guardians. The team was not able to randomly select participants due to the need for service providers to contact potential participants in advance of the date the FGD was held, their availability on that date, and the time-constrained field work. Self-selection bias also was possible from those who agreed to participate, and so there may be an overrepresentation of positive responses about OVC service delivery. There also could be an overrepresentation of positive responses about PACT’s capacity building due to potential interest in receiving further support from PACT, USAID, PEPFAR or the U.S. Government in the future. The team tried to mitigate this type of selection bias through the anonymous online survey and confidential one-on￾one KIIs. Recall bias also was possible by participants due to inaccurate memory about activities and implementers. Results from a sample of KIIs and beneficiary FGDs may give an indication of capacity changes, but not necessarily attribute those changes to PACT’s support. For example, a number of Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 4 the organizations interviewed received capacity building from other organizations and individuals outside PACT. Moreover, FGDs and KIIs did not allow for feedback on the change since program inception, because all the participants were not with the program since the beginning of PACT’s support and some sub-partners have not been operating in the same area or implementing the same services since 2010. The time available for research did not allow independent observation of service delivery. Thus independent verification of quality and dosage was not possible. In addition, “dosage” and “exposure” vary as the sub-partners implement different services, some sub-partners implement many different services, and some have received more PACT support than others. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Q1: PACT’s progress on organizational capacity building of sub-partners: Finding: PACT has made progress in effectively building the organizational capacity of 7 of the 8 sub-partners sampled in the evaluation, using factors such as regular mentoring support after workshops; constant monitoring; supporting both organizational and professional growth; and remaining flexible in addressing organizational needs. The lack of progress seen in one organization is due to weak leadership in that organization which led PACT to focus on specific activities only and not on addressing the real needs of the whole organization, at the level of intensity that was needed. Conclusion: PACT provides a good, well-considered and appropriate range of capacity-building services to its sub-partners in Swaziland. However, some important organizational training and support should have been introduced earlier in the project, such as organizational development (OD), leadership and governance training and related activities. Recommendation: PACT should continue OD capacity building by working with sub-partners’ boards and senior management to further develop their understanding about governance, roles and responsibilities, management and leadership, and assist the sub-partners to finalize succession planning as they develop and refine their organizational charts. Q2: PACT’s progress on HIV/AIDS and OVC technical capacity building of sub￾partners: Finding: PACT has made good progress in building the technical capacity of its sub-partners in HIV prevention and OVC services and has provided some programmatic technical assistance in treatment/care. All 8 sampled sub-partners scored higher in their technical capacity assessments after receiving technical capacity building from PACT since 2011. Conclusion: PACT’s focus on building capacity and backstopping of gender-focused services and children-focused services (including OVCs) has been a strength, with about 75% of its sub-partners reporting that their organizations benefitted from PACT’s technical capacity building on HIV/AIDS. Recommendation: Pact should continue with its technical capacity-building strategy, but consider more regular technical capacity-building workshops. External assessments by experts would augment self-assessments as a more objective gauge of quality and identify any additional capacity-building needs. Q3: Improvements in quality, targeting and dosage of sub-partners’ services through PACT’s support: Finding: According to most respondents from sub-partner organizations, beneficiaries, and government, the quality, targeting, and frequency of sub-partner services has improved based on PACT’s support, including monitoring and mentoring visits. Although there have been increases and Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 5 decreases in targets and results over time across sub-partners, those delivering care and treatment services have shown the greatest increases in reach Conclusion: The coverage of some services has increased through expanded targeting by sub￾partners, as has the focus on increasing the dosage and frequency of service provision via PACT’s support. The sub-partners providing care and treatment services have shown the highest results against their targets in 2012-2013. Improvements in service quality could not be independently verified but many beneficiaries, sub-partners, and government staff reported that the quality of many services delivered by the sub-partners has improved. Recommendation: PACT should continue emphasizing quality service provision by sub-partners as the need for the delivery of high-quality HIV/AIDS services in Swaziland continues. Likewise, PACT’s emphasis on strengthening sub-partners’ service dosage and frequency should be maintained. In the next round of funding USAID should examine whether an increase in funding for staffing and service delivery would enable sub-partners capable of efficiently managing funds to deliver services to a greater number of beneficiaries in Swaziland. Q4: Impact of geographic coverage on the quality and dosage of services and alignment with national strategies: Finding: Determining how different coverage of sub-partners’ activities impacts on the quality and dosage of services was not possible as dosage varies by intervention, although concentrating services in one geographic area with appropriate funding may lead to greater frequency of beneficiary access. Service quality is affected by the number and quality of skilled personnel and the use of quality service standards. PACT’s sub-partners have been trained to align their services to the national standards and PACT’s support has helped them participate in government planning structures and in national strategy development processes. Conclusion: There is no evidence to show that increased geographic coverage results in a reduction in the quality of services. The emphasis put on dosage by PACT has improved some sub￾partners’ service dosage and frequency, especially in prevention- and OVC-focused services. All eight sampled sub-partners were able to demonstrate that their work as aligned to the National Strategic Framework (NSF) on HIV/AIDS, and the sub-partners providing OVC services use the OVC Quality Service Standards (QSS), which were initiated and developed by PACT and other organizations. Recommendation: PACT should assist sub-partners to focus on new ways of incentivizing staff (to facilitate them to stay longer in their jobs) as well as succession planning as these factors both affect the coverage and reach of services. PACT should encourage its sub-partners to prioritize participating in planning and technical working groups (TWGs) to help support the development of high-quality national strategies and systems in the future and continue to make referrals to government services and other NGOs. Q5: Approaches working well for delivering sustainable community services for OVC: Findings: All 4 OVC organizations supported by PACT to deliver OVC services have a multi￾sectoral approach to supporting OVC, but each has a unique implementation approach and a unique package of services offered to their beneficiaries and constituents The OVC sub-partners motivated that funding must continue for most of the listed OVC service approaches. Pact’s work with government helps to ensure that OVC services are a central priority within of NERCHA and the DSW’s work. Pact sits on the relevant government committees and proactively Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 6 keeps the debate around OVC services at the forefront, as it did when helping to develop the OVC Standards. Conclusions: The approaches and OVC services working well include: Early childhood and care development at care points enhancing OVC access to primary schools; improving access for OVC to health and other services through transportation provision; registering births and acquiring of certificates to enable more OVCs to attend school; using innovative agricultural technologies, such as permaculture, to improve nutrition at schools and at homesteads; supporting savings and credit groups to help families save and to strengthen communities; providing life-skills training for young people to increase their awareness of needs and wants; education on child rights leading to increased local condemnation of child abuse and increased medical and psycho-social support (PSS) of abuse survivors. Recommendations: PACT and its sub-partners should continue involving community stakeholders, building trust and tackling basic needs of OVC and their caregivers. The OVC sub￾partners should increase linkages with other organizations so that more services are brought to the community. Q6: Development of national standards and processes to improve the OVC response: Finding: PACT was an initiator of Swaziland’s OVC standards, a partner in finalizing the NSF on HIV/AIDS, a supporter of the Early Learning and Development Standards, and a member of TWGs on OVC, PSS, child protection, and Early Child Care and Development (ECCD). Conclusion: PACT initiated, piloted and developed quality standards for OVC services that are positively affecting the quality of service delivery, and PACT’s work with government has influenced and empowered the sub-partners, positioning them well to continue to work with government when REACH ends. Recommendation: PACT should continue its work with government departments until the end of REACH. Initiating and responding to the development of standards and other national processes relating to children, especially OVCs and HIV/AIDS, will be a legacy of PACT’s and USAID’s support to the Government of Swaziland. PACT’s work on introducing mHealth technology to its sub￾partners has been an innovation and should be continued. Q7: CANGO’s progress in becoming an umbrella grants mechanism: Finding: The capacity of CANGO to operate as a UGM has increased significantly based on PACT’s capacity building of the organization through intensive and personalized training, support, mentoring, peer reflection, and re-training, although support needs to continue. Conclusion: PACT’s support of CANGO to become a UGM is producing results, but CANGO cannot yet operate as an UGM on its own. Moreover, its human capacity needs to be expanded and stabilized. Recommendation: There is need for a clear division between capacity building activities related to CANGO’s internal capacity as an organization versus CANGO’s capacity as a UGM. To enable CANGO to become a functioning UGM as quickly as possible, senior CANGO staff need increased mentoring so that leadership can emerge quickly, for succession planning, to foster its organizational independence, and to make staff aware of the strategy and timelines going forward Q8: Strengths and weaknesses of PACT’s engagement with CSO recipients of Global Fund resources: Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 7 Finding: PACT’s capacity building in a few of the Global Fund capacity requirements has been an advantage to those sub-partners who have received this training, including Global Fund super sub￾recipient CANGO. But PACT has not taken a comprehensive approach in building the capacity of its sub-partners to become potential Global Fund sub-recipients although its interactions with NERCHA have helped to foster CANGO’s relationship with the Global Fund sub-recipient. Conclusion: PACT’s liaising with NERCHA on Global Fund issues has had some positive effects on civil society organizations, but there is an even greater need for liaising with NERCHA and advocating on behalf of civil society organizations for future funding. In addition, CANGO needs more strengthening of its Global Fund capacity requirements as a super sub-recipient, especially in the areas of procurement/supply management and risk management. Recommendation: PACT should prioritize ensuring CANGO’s involvement in the negotiations leading up to and the drafting of the concept note to the Global Fund, emphasizing the need for funding to civil society sub-recipients through the Global Fund’s new funding model (NFM) in 2015. CANGO and other sub-partners would benefit from additional capacity building in Global Fund capacity requirements if resources are available to do so, especially in the areas of procurement and supply management and risk management. Overall Conclusions and Recommendations Overall Conclusion: Each sampled organization underscored the importance of PACT’s organizational capacity building to their present stability and all were interested in continuing support from PACT in their organizational development. Most would also like additional HIV/AIDS technical capacity building in the future, at least some refresher training. As the HIV epidemic continues to have a heavy impact in Swaziland, further strengthening of the overall response to the epidemic by civil society organizations is needed. Overall Recommendation: While the need for building the organizational capacity of CSOs in Swaziland continues, there is more urgent need for a scaled up response to the HIV/AIDS epidemic in Swaziland resulting in an “AIDS-free generation”. Expanding the HIV/AIDS technical skill set and beneficiary reach of CSOs in Swaziland should be the priority for USAID and PEPFAR resources over the next five years. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 8 EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose The primary objectives of this mid-term performance evaluation of PACT’s Community Reach Project in Swaziland were to:  assess the quality of project implementation;  assess progress that has been made by the project toward achieving its set goal, objectives, expected outputs and/or outcomes;  determine which approaches and activities are working well and why, and  indicate areas that may require mid-term corrections. In terms of funding volume, PACT is PEPFAR/Swaziland’s largest partner in its OVC portfolio. Thus, this evaluation also includes a focus on PACT’s effectiveness in strengthening technical and institutional capacity for quality OVC service delivery. Additionally, findings and recommendations from this evaluation are meant to inform future USAID solicitations in the area of local institutional and technical capacity building, as well as the area of OVC. The audience of the evaluation report is USAID and PEPFAR/Swaziland, PACT as implementing partner, the CANGO as the local Umbrella Grants Management Partner, USAID Southern Africa, and the Global Health Technical office in USAID/Washington. Evaluation Questions The eight key evaluation questions to be answered by this mid-term evaluation are: 1. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? 2. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? 3. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? 4. How different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? 5. What approaches for delivering sustainable community services for OVC are working well? 6. To what extent has PACT strengthened efforts to improve the OVC response beyond sub￾partner service delivery through activities related to developing of standards and other national processes? Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 9 7. What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? 8. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? Each of these questions is explored in detail in the section on Findings, Conclusions, and Recommendations beginning on page 20. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 10 PROJECT BACKGROUND Community REACH 2 in Swaziland is a five-year project (March 2010 to March 2015) managed by PACT and funded by USAID/Swaziland. REACH 2 is a successor project to Community REACH I (2005 to 2010) which provided grants to organizations providing HIV services in Swaziland. The current award continued with activities under the previous agreement and added new activities with capacity building of a local organization, CANGO, to become an Umbrella Grants Manager for civil society organizations. The REACH program was designed to facilitate the efficient flow of grant funds and to deliver targeted technical assistance and capacity building services to organizations contributing to the HIV/AIDS response. The overall goal of REACH is to reduce the impact of HIV & AIDS and improve health care for the people of Swaziland by developing a strengthened, coordinated civil society response to HIV & AIDS. In order to achieve this goal, the project is expected to achieve the following results: Result 1: To strengthen 1-2 local NGOs in umbrella grant management thereby enabling them to manage and sub-award funding from PEPFAR, Global Fund and other major donors in Swaziland. Result 2: To provide PEPFAR funding and minimal technical assistance to 3-5 mature NGOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 3: To provide PEPFAR funding and a full package of technical assistance to 10 – 15 nascent NGOs/CBOs/FBOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 4: Provide an as yet to be determined number of Global Fund recipients with training, mentorship and linkages to appropriate technical assistance. USAID/Swaziland supports programs focusing primarily on prevention (sexual prevention, PMTCT, male circumcision), impact mitigation with a focus on vulnerable children and human and institutional capacity development (capacity building for community-based programming, strengthening of health systems, institutional strengthening of local NGOs). Because the HIV response in Swaziland is seriously hampered in all program areas by limited human and institutional capacity, the PACT award was designed to address these capacity gaps among civil society organizations providing community services. PACT operates as an umbrella grant mechanism (UGM) with a focus on strengthening the organizational effectiveness of NGOs to improve their capacity in project management, financial accountability, and monitoring and evaluation for results-oriented programming and organizational sustainability. PACT’s mandate is to build the capacity of organizations in a number of critical areas including grants and finance management, technical and program management, organizational development, monitoring, evaluation, reporting and learning. PACT uses capacity assessments to conduct in-depth reviews of each partner’s strengths and weaknesses in defined areas, based on which an institutional strengthening plan (ISP) is developed which details planned interventions to boost capacity in specific areas. PACT staff provides on-site assistance to address organizational weaknesses as well as customized group trainings to address institutional capacity needs identified across partner organizations. PACT started REACH 2 in March 2010 with 14 partners who were existing partners under REACH 1. PACT officially transitioned 13 of the 14 partners to REACH 2 in October 2010 while the Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 11 preparations for a new Request for Applications (RFA) were ongoing. One partner, Family Life Association of Swaziland (FLAS MC), was closed out in October 2010. The RFA for implementing partners was issued in December 13, 2010 and it resulted in 12 partners starting new grants in October 2011 (FY12). Of the 12 partners who successfully competed in the RFA process, 8 were partners from REACH 1 and 4 were new to Pact (e.g. Good Shepherd Hospital (GSH), TechnoServe Swaziland (TNS), Lutheran Development Service (LDS), and Cheshire Homes of Swaziland). The partners who did not secure a place in the new project were closed out at the end of FY 2011. These included FLAS AI, Action Against Hunger, World Vision, Roman Catholic Church, and Nhlangano AIDS Training Information and Counseling Centre (NATICC). Khulisa Umntfwana and Lutsango LwakaNgwane, both organizations that seek to support Swazi culture and traditions, were brought into the REACH 2 project as partners through a non-competitive process by recommendation from USAID in April 2011 to address gender issues through USAID’s Gender Challenge Fund. PACT currently supports 14 partners delivering HIV services in various technical areas which include sexual prevention, HIV testing and counseling, care and treatment, TB/HIV, impact mitigation, and gender. From FY2011 – FY 2013, PACT partners reached over 80,000 beneficiaries, mostly with OVC and HIV care services (Figure 1). Some of PACT’s sub-partners operate in all four of Swaziland’s administrative regions while others have a more limited reach (see Table 1 below). Figure 1. Beneficiaries reached by PACT sub-partners by HIV program (2011-2013) PACT also supports the Coordinating Assembly of NGOs (CANGO) in developing its capacity to serve as a local UGM for civil society organizations in Swaziland for other donors, with a specific focus on the Global Fund under the National Emergency Response Coordinating Agency (NERCHA). NERCHA lists CANGO as a ‘super-sub-recipient’ tasked to serve as an umbrella body for civil society organizations. Five PACT sub-partners have shifted over the last 15 months to be managed directly by CANGO (see Table 1 below) with close mentoring and supervision from PACT. - 20,000 40,000 60,000 80,000 100,000 120,000 No. Beneficiaries reached by PACT Subgrantees (FY2011-FY2013) 2011 2012 2013 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 12 In FY2013 PACT supported CANGO to undertake a competitive selection process for three new sub-partners, which started to receive funding in August 2013. As PEPFAR/Swaziland’s largest OVC partner, PACT has made significant investments in the OVC response not just through sub-partners, but also through strategic national level efforts, such as the development of standards (OVC service standards, Early Learning Standards) as well as participation in national technical working groups and processes. Table 1 summarizes the sub-partners’ technical areas and geographical focus and shows the current rating of each NGO per PACT’s capacity development stage rating, as defined in Table 2: Table 1. Sub-partner Organizations Funded by PACT and CANGO Sub-partner Organizations Fiscal Years Funded by REACH 2 Technical Areas Regional Coverage Capacity Stage & Rating (1-5) Funded directly by PACT 1. Swaziland Business Coalition on Health and AIDS (SWABCHA) FY2011-2013 3 years Prevention, Gender, HCT, TB/HIV Hhohho, Manzini Expanding (4) 2. Swaziland National Network of People Living with HIV&AIDS (SWANNEPHA) FY2011-2013 3 years Treatment Hhohho, Manzini, Lubombo, Shiselweni Marginal (3) 3. The Salvation Army (TSA)* FY2011-2013 3 years HCT, Care & Support, TB/HIV Hhohho, Manzini Expanding (4) 4. Cabrini Ministries* FY2011-2014 4 years Treatment, Paediatric Treatment, Care & Support & Paediatric Care & Support; HCT, OVC, Prevention Lubombo Expanding (4) 5. Good Shepherd Hospital (GSH) FY2011-2014 4 years Treatment, Paediatric Treatment, Care & Support & Paediatric Care & Support Lubombo Expanding (4) 6. Lutsango LwakaNgwane FY2011-2014 4 years Gender/Sexual Prevention Manzini Emerging (2) 7. TechnoServe Swaziland FY2011-2013 3 years OVC n/a Mature (5) 8. Lutheran Development Service (LDS)* FY2011-2014 4 years OVC Lubombo Marginal (3) 9. Coordinating Assembly of Non￾Governmental Organizations (CANGO) FY2012-2014 3 years Umbrella Grants Management (UGM) n/a Emerging (2.89) Funded through CANGO 10. Save the Children FY2011-2014 OVC, Gender, Hhohho, Manzini, Marginal (3) Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 13 Sub-partner Organizations Fiscal Years Funded by REACH 2 Technical Areas Regional Coverage Capacity Stage & Rating (1-5) Swaziland (SCSWD) 4 years Prevention Lubombo, Shiselweni 11. Voice of the Church (VOC)* FY2011-2014 4 years Prevention AB Hhohho, Manzini, Lubombo, Shiselweni Marginal (3) 12. Bantwana Initiative FY2011-2014 4 years OVC & Gender Lubombo Expanding (4) 13. Khulisa Umntfwana FY011-2014 4 years Gender/Sexual Prevention Manzini Emerging (2) 14. Cheshire Homes of Swaziland FY2011-2014 4 years Prevention, Care & Support Manzini, Lubombo, Hhohho Marginal (3) 15. Joyful Heart Aug. 2013-2014 1 year HCT, Treatment, TB/HIV Lubombo Emerging (2) 16. Kudvumisa Aug. 2013-2014 1 year HCT, Treatment, TB/HIV Lubombo Emerging (2) 17. Super Buddies Aug. 2013-2014 1 year Prevention Manzini, Shiselweni Emerging (2) *Faith-based organization Table 2. PACT’s Capacity Development Stage Rating Nascent All capacity areas measured are in rudimentary form and the organization does not yet have systems or processes in place. The organization may be flexible and full of energy. Emerging Some capacity areas measured show development of systems, policy, and vision of labor. Marginal Most capacity areas are reinforced with clear and documented policies, practices and systems. The organization is beginning to develop a track record of implementation and is starting to engage in participatory processes for change management. Expanding Capacity areas measured demonstrate a track record of achievement, accountability, participation throughout the organization as well as a mixture of flexibility and strong systems. Mature All capacity areas measured demonstrate sustainability as well as successful collaboration with a mixture of other organizations to ensure maximized impact. EVALUATION METHODS & LIMITATIONS This Performance Evaluation utilized a non-experimental design that excluded a rigorously-defined counterfactual or comparison group. However, the evaluation team incorporated before-after comparisons to determine changes in project activities, outputs and outcomes over time. The team employed a mixed methods approach (collection of both qualitative and quantitative data) to collect both primary and secondary data and to ensure that the eight evaluation questions were answered comprehensively. The evaluation’s conceptual framework (Figure 2) involved examining PACT capacity building processes (inputs), internal organizational changes among the sub-partners (outputs), and changes in Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 14 the sub-partners’ delivery of services (outcomes). Assessment of outputs and outcomes focused on elements of capacity (systems/policy, organizational, individual) as recently defined by PEPFAR’s Capacity Building Framework1. Examining these elements of capacity and capacity building allowed the evaluation team to determine the Effectiveness (in achieving outcomes, meeting standards), Efficiency (in delivering services and increasing reach using a minimum of inputs), Relevance (in engaging target populations and promoting learning), and Sustainability (mobilizing resources, increasing legitimacy) of REACH interventions. Figure 2. Conceptual Framework for Evaluation of Capacity Building Sampling Framework There are 17 sub-partners affiliated with PACT’s REACH project, including 4 for which funding ended in September 2013. A subset of 8 organizations plus CANGO (for a total of 9 organizations) were selected for in-depth interviews and external technical capacity assessments. The 4 criteria that were used for selecting the sample are as follows: Table 3. Criteria for selecting the sample of partners to be interviewed Criterion No. of Orgs meeting criteria Result Names of Organizations Selected CANGO 1 CANGO is included in the sample 1. CANGO Organization offers OVC services 4 All 4 organizations offering OVC are included in the sample 2. Bantwana Initiative 3. Cabrini Ministries 4. Lutheran Development Service (LDS) 5. Save the Children Swaziland (SCSWD) A networking 2 Both the networking 6. Swaziland Business Coalition on 1 PEPFAR Capacity Building and Strengthening Framework. 2012. http://www.pepfar.gov/documents/organization/197182.pdf Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 15 Criterion No. of Orgs meeting criteria Result Names of Organizations Selected organization organizations are included in the sample Health & AIDS (SWABCHA) 7. Swaziland National Network of People Living with HIV&AIDS (SWANNEPHA) Number of beneficiaries reached as of Sept 2013 2 Of the remaining organizations, the final two have a high number of beneficiaries and so were selected. 8. Khulisa Umntfwana 9. Voice of the Church (VOC) Another criterion used to select the sample was to include a range of length of experience the organizations had with the REACH Project. The 9 organizations chosen for the sample have varied length of experience with the REACH Project as shown below: Table 4. Sampled Organizations’ Length of experience with REACH project Year started with REACH No. Organizations in Sample 2006 2 2007 0 2008 2 2009 2 2010 0 2011 2 2012 1 TOTAL 9 All the sub-partner organizations working with OVCs were selected for the sample. Data collection in these organizations included separate focus group discussions with OVC beneficiaries aged 12-18, caregivers, and parents/guardians at ten randomly-selected sites where the OVC services are offered. Four sites were chosen from SCSWD, one in each region where they work; three sites were chosen from Bantwana; two sites were chosen from LDS; and one from Cabrini. Due to time constraints, it was not possible to select OVC FGD participants at random. Therefore, the relevant sub-partner was informed of the selected site and the date of the FGD after which they organized a group of participants to be present. Key informants for interviews were chosen to include a selection of managers and program officers from the 9 sampled sub-partners; relevant staff at PACT’s HIV/AIDS and OVC organizational partners in Swaziland as listed in its cooperative agreement with USAID in March 2010; and relevant USAID/PEPFAR staff members. All 17 organizations, as well as several organizations previously funded by PACT, were invited to answer the anonymous online survey to obtain their views on the capacity building support provided by PACT. Methodology Various methods were employed in this Performance Evaluation: Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 16 a) Document and data review; b) Key informant interviews (KIIs) with PACT managers and staff; a selection of the nine sampled sub-partner managers and staff; and relevant stakeholders, i.e. Swaziland National AIDS Programme (SNAP), National Emergency Response Council on HIV and AIDS (NERCHA), National Children’s Coordination Unit (NCCU), Ministry of Education & Training, UNICEF and USAID; c) Focus Group Discussions (FGDs) with male and female beneficiaries, parents and caregivers affiliated to OVC sub-partner programs in 10 sites; and FGDs with the selection of the eight sampled sub-partner managers and staff plus CANGO; d) Technical Capacity Assessments with the eight relevant sub-partners; e) Anonymous online survey of staff and managers for all 17 sub-partners. The design of the KII, FGD, and the online survey tools was guided by the eight evaluation questions. Annex III provides the evaluation tools used by the team. DATA COLLECTION Document and Data Review Document review provided the team with key background and contextual information and helped identify any data gaps. The evaluation team used existing documents along with the results of fieldwork to clarify the project’s underlying theory of change, specifically the outputs that were directly attributable to project activities, as well as the outcomes toward which the project contributed (but for which it is not solely responsible). The key documents and data that were reviewed are listed in Annex IV. Key Informant Interviews (KIIs) In-depth, semi-structured, key informant interviews, using themes derived from the 8 REACH evaluation questions, were conducted to obtain perspectives on: the quality of the capacity building processes, the extent to which the desired system/ organizational/ individual changes have occurred, and the identification of gaps in sub-partner needs. The KIIs incorporated a few questions styled around Most Significant Change to try to capture the extent of change since the inception of the project. The KIIs were conducted with various staff within the sample of 8 sub-partner organizations and CANGO (CEO or Head of Organization, Finance Manager, M&E Manager; and Program Manager). In addition, KIIs were conducted with representatives from:  USAID/Swaziland managers  PACT/Swaziland managers  Department of Social Welfare (DSW)  National Children’s Coordination Unit (NCCU)  MOH/AIDS Programme (SNAP)  NERCHA  UNICEF Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 17 Focus Group Discussions (FGDs) Focus group discussions were conducted with mangers and staff in the eight sampled sub-partner organizations plus CANGO. These FGDs used questions derived from PACT’s Organizational Capacity Assessment Tool (OCAT), Management Control Assessment Tool (MCAT), and Monitoring, Evaluation and Reporting Tool (MER-OCAT), and a few general questions related to the REACH evaluation. Separate focus group discussions (FGDs) with OVC beneficiaries ages 12-18, caregivers, and parents were conducted in ten sites where OVC services are delivered by the 4 relevant sub-partners. These FGDs utilized a semi-structured guide to ensure comparable information was collected at each site. The FGDs focused on their perspectives of the quality and extent of OVC service delivery, program management, and any gaps identified. Prior to conducting the FGDs on OVC services, a Performance Scorecard was completed by each FGD participant on the satisfaction with and frequency (i.e. dose) of OVC service delivery categories. The scorecard was refined to ensure adequate quantitative indicators of the OVC services to reflect the range of respondents’ views. As an incentive for participation, the team provided parent and caregiver focus group discussants with a R30 air time voucher and OVC beneficiaries with school supplies of comparable value. Technical Capacity Assessments At the head offices of the 8 sampled sub-partners, the relevant Technical Capacity Assessments were administered with the individuals selected for the KIIs. The use of this tool allowed triangulation with the sub-partners’ previous Technical Capacity Assessment scores around the quality of service delivery and any changes since the Technical Capacity Assessments were administered in prior years. Anonymous Online Survey An online survey sent to PACT’s present and past sub-partners (16 sub-partners and CANGO) explored their satisfaction with the quality and utility of the project’s organizational and technical capacity building efforts and the effects on organizational functioning and services delivery. DATA ANALYSIS The methodology described above generated a mix of qualitative and quantitative data which was analyzed using methods appropriate to each. Some of the techniques which the evaluation team used to analyze the data based on the 8 evaluation questions are briefly described below.  Theme analysis, which involved: o Viewing the data several times as a whole o Identifying patterns and themes o Reorganizing the data (e.g. coding the data according to the themes identified).  Triangulation: o Cross-checking the data in order to increase the confidence in the findings. o Use of multiple data sources (PACT and sub-partner managers and staff as well as beneficiaries) o Use of multiple data collection methods (document review, Technical Capacity Assessments, KIIs, online survey and focus group discussion). Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 18  Descriptive Statistics including graphs: o Trend Analyses o Frequencies o Percentages o Composite Scores, where applicable The cleaned and coded qualitative and quantitative datasets will be shared with PEPFAR. Response Rates Overall, the evaluation reached or exceeded the targeted number of respondents except for the online survey which had a relatively low response rate compared to the other data collection methods. Table 5. Overall Response Rates in the Evaluation Data Collection Method No. Assessments No. Participants Target Actual Target Actual Response Rate Key Informant Interviews 45 60 133% Focus Group Discussions (OVC) 10 10 240 231 96% Focus Group Discussions (8 Sub-partners) 8 8 32 35 109% Technical Capacity Assessments 8 8 32 35 109% Anonymous Online Survey 64 35 55% TOTAL 26 26 413 396 96% Table 6. Response Rates in the Anonymous Online Survey Category of Respondents Targeted Actual Response Rate CEO, MD, or head of organization 16 7 44% Financial Manager 14 6 43% M&E Officer 13 9 69% Program Officer 13 6 46% Other 8 7 87% TOTAL 64 35 55% Limitations By their nature, qualitative evaluations of policies and program implementation such as this evaluation allow examination of the broader context, rather than specific measurements of compliance to policy elements or programmatic guidelines. This evaluation’s extremely broad scope (1 primary grantee focusing on organizational development, 17 sub-partner organizations implementing a wide range of HIV/AIDS and OVC interventions, 8 research questions and 6 sub-questions) resulted in lengthy interviews and focus group discussions with respondents who often had time constraints. Consequently, many respondents gave only cursory information around the issues that we sought to explore more deeply. Potential bias was possible from all respondents. Selection bias from the OVC beneficiaries, caregivers, and parents/guardians is feasible as the evaluation team could not randomly select participants and depended on sub-partners to contact potential participants in advance of the FGD Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 19 date. As such the sample was based on the participants’ availability on that date. Given sub￾partners’ potential interest in receiving further support from PACT, USAID, PEPFAR or the U.S. Government in the future, there could also be an overrepresentation of positive responses about PACT’s capacity building. The team tried to mitigate this selection bias through the anonymous online survey and confidential one-on-one key informant interviews. Recall bias also was possible by participants due to inaccurate memory about activities and implementers. Results from a sample of KIIs and beneficiary FGDs may give an indication of capacity changes but not necessarily attribute those changes to PACT’s support. For example, a number of the organizations interviewed received capacity building from other organizations and individuals outside PACT. Moreover, FGDs and KIIs may not have allowed for feedback on the change since program inception, because many participants were not with the program since the beginning of PACT’s support and some sub-partners have not operated in the same area or implemented the same services since 2010. Capacity assessment data of PACT sub-partners was made available for the years 2011-2013. But due to time constraints, the evaluation team could not re-administer the 4 PACT assessment tools as planned and in the same manner as implemented by PACT for purposes of validating and comparing the PACT data to the evaluation data. Consequently, the team administered only the Technical Capacity Assessment Tool, and the quantitative data set available for time series analysis was largely limited to the pre-existing 2011-2013 data. The absence of a control or comparison group to use as a counterfactual means that changes in organizational and technical capacity cannot be attributed to only PACT’s capacity building efforts, and increases and decreases in capacity development between 2011 and 2014 could not be objectively verified by the team. The online survey data set was limited by the number of respondents to the online survey. Lastly, the time available for research did not allow independent observation of service delivery. Thus independent verification of quality and dosage was not possible. In addition, “dosage” and “exposure” vary as the sub-partners implement different services, some sub-partners implement many different services and some have received more PACT support than others. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 20 FINDINGS The findings below are organized along the 8 evaluation questions. Each question presents the main findings, conclusions and recommendations specific to the evaluation question. Overall conclusions and recommendations are presented at the end of the report. Overview of PACT’s Capacity-Building Approach “The major objective of the program is to build capacity of local organizations to effectively contribute to the response against HIV/AIDS...The overall program goal of the program is to reduce the impact of HIV and AIDS and improve health care for the people of Swaziland by developing a strengthened, national level, coordinated civil society response to HIV and AIDS in Swaziland.” FY2013 Semi-Annual Progress Report, covering the period October 1, 2012 to March 31, 2013 PACT’s global website shows that its capacity building approach consists of:  Training/workshops;  Mentoring/coaching;  Peer education;  Information and resources referral;  Consulting services; and  Sub-grants In Swaziland, PACT primarily offers training workshops and mentoring as its main means of building organizational and technical capacity. It also, on occasion, encourages co-facilitation by peer organizations in the capacity building workshops. In addition, PACT develops the capacity of a local civil society organization, CANGO, to become an UGM through workshops and mentoring. CANGO is also monitored to manage sub-grants and to provide information to sub-partners. A presentation by, and literature from, PACT/Swaziland, shows baseline assessments of the sub￾partners/organizations is the first step in capacity building. This measure identifies the gaps in the organizations’ processes and structure and sets the stage for a support intervention. The tools that assess organizational capacity, known as Organizational Capacity Assessments (OCAs), are applied by PACT in a collective and collaborative manner with inputs from all staff at the organizations, who rate their performance in areas on a scale of 1 (low) to 5 (high) and then agree on a final consensus score. The four assessment tools are:  OCAT - for general Organizational Capacity Assessment;  MER-CAT - for general Monitoring and Evaluation Capacity Assessment;  MCAT - for general Management Capacity Assessment; and  Technical Capacity Assessment - for specific services offered by organizations. These tools are applied annually and form the basis from which PACT’s Institutional Strengthening Plans (ISPs) are developed for each organization. A collaborative approach was used in administering the tools, in that the organizations collectively and under guidance from PACT, score themselves on all the tools. The ratings are discussed with PACT, which scores the organizations separately, and a discussion is held with the organizations to arrive at a final agreed consensus score. This collaborative approach is a central theme for PACT and underpins their work with NGOs. Above and beyond the OCA tools, PACT uses the Organizational Performance Index (OPI) to assess performance. The OPI defines organizational performance through four domains, each with two sub-components: Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 21 1. Effectiveness – Achieving Results and Meeting Standards 2. Efficiency – Delivering Services and Increasing Reach 3. Relevance – Engaging Target Populations and Promoting Learning 4. Sustainability – Mobilizing Resources and Increasing Social Capital As reported in PACT’s FY2013 Annual Progress Report, PACT has integrated the OPI measurements into the annual participatory capacity assessment process, resulting in scores in four overarching domains to which the sub- partners can refer. Figure 3 shows how all partners have fared across the 4 OPI domains between FY2011 and FY2013. In examining trends in the 4 OPI domains, sub-partners show a slower improvement rate in the Effectiveness domain compared to the other 3 domains. One reason for this apparent slower progress may be in the level statements for Effectiveness, which at the highest level (i.e. Level 4) are:  Achieving results – Level 4: The organization has met over 75% of outcome level targets for its programs & services.  Meeting standards – Level 4: The organization consistently meets existing standards and is involved in setting new national and/or international standards that govern their programs & services. Figure 3. Comparison of Average OPI Scores for all sub-partners excluding new partners FY2011- 2013 Source: PACT data To achieve the highest level in Achieving Results requires evaluations (rather than routine reporting) of outcome level changes in behavior and/or the enabling environment. Most sub-partners are unable to do this on an annual basis. To achieve Meeting Standards is also challenging for most sub￾partners, as they generally do not working directly with the government on standards; rather PACT is the main organization that works on behalf of civil society with government departments, which raises a question as to whether this measure is achievable by all the sub-partners. In comparison, sub-partners are able to achieve the highest levels of Efficiency. Efficiency is about delivering services and increasing reach. This is the essence of the work of the NGOs and is a central focus for them. PACT’s capacity building support for monitoring and evaluation (M&E) is 1.94 2.00 2.00 2.00 2.35 3.15 3.00 3.08 2.42 3.46 2.92 3.19 0 1 2 3 4 Effectiveness Efficiency Relevance Sustainability OPI Score (1-4) Average OPI score for partners FY11-FY13 OPI Score FY 11 OPI Score FY 12 OPI Score FY 13 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 22 largely concerned with this area and provides a strong level of support to organizations to report in this domain. For the domain of Relevance, there has been a drop in the rating from 3 in 2012 to 2.92 in 2013. PACT’s explanation for this is that some partners do not engage with their stakeholders/ beneficiaries in the planning of the activities they will implement. With regard to the fourth and final domain, Sustainability, all sub-partner organizations will be involved in trying to secure other funding to sustain themselves as the end of the REACH project draws near. Those partners that have already set up partnerships with other funding organizations appear to be more confident about their ability to survive; for example, Bantwana is supported in part by World Education, which is a US-based organization, and Cabrini has donors in Swaziland and also in the USA. The Institutional Strengthening Plan (ISP) is the action plan to improve the organization based on weaknesses or gaps identified in the various capacity assessments and assessment discussions. The ISP has clearly defined areas, namely:  Governance and Leadership;  Management Practices;  External Relations and Partnerships;  Sustainability;  Grants and Finance;  M&E; and  Programs/Technical The first four areas relate to organizational development (OD) capacity building and are the focus of the OCAT. The other areas are dealt with under other relevant tools, including the MER, MCAT and Technical Capacity Assessment. Under each ISP area, capacity gaps, prioritized action plans, timeframe, responsibility and status are recorded. The ISPs are reviewed on a quarterly basis to see what and how progress has been made. This method and approach applies to all sub-partners with the exception of CANGO, which is being developed as a UGM. CANGO, before taking on the role of UGM, was subject to the same assessment process as other sub-partners. It now is assessed using the OD Roadmap, a tool that specifically looks at how an organization is able to support and strengthen others. This is in line with the roles and responsibilities of becoming a UGM. The OD Roadmap was adapted for use in Swaziland and has its roots in Namibia, where it was used to assist the graduation of civil society projects from receiving PACT funding to a situation where they could apply for and get direct USAID support. It was piloted in Cambodia, Swaziland, Thailand and Zimbabwe, and the pilot was adapted for the REACH project. The Roadmap consists of ten discrete areas that cover functions of an organization and the additional section is concerned with what CANGO has to do to be a fully effective UGM. The Roadmap requires the staff members of CANGO to self-assess their organization in various categories and then agree via consensus on a rating score, similar to the OCAT process. This process can take several hours, to a day, for dialogue, deliberations and consolidation of the rating. The findings under Question 7 provide further comments on the OD Roadmap. In the Findings below, Question 1 addresses all the organizational capacity building tools and areas, and Question 2 reflects on the Technical Capacity Assessment Tool and development of sub￾partners technical capacity. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 23 Question 1. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? FINDINGS PACT conducts annual assessments of its sub-partners using its organizational capacity-building tools. Figure 4 shows the trends in Organizational Capacity Assessment Tool (OCAT) scores for 2011- 2013 for the sub-partners. There are two sampled organizations which are not progressing well, LDS, and SWANNEPHA. Additional mentoring has been given to LDS for the remaining time of their funding from PACT. PACT’s support and funding to SWANNEPHA ended on September 30, 2013. Figure 4. Overall OCAT results for PACT sub-partners 2011-2013 Source: PACT data - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 Governance and Leadership AVERAGE SCORES OCAT Assessment Results (PACT) Bantwana Cabrini Cheshire GSH Khulisa LDS Lutsango SCSWD SWABCHA SWANNEPHA TSA VOC AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 24 Figure 5. Governance and Leadership OCAT results for PACT sub-partners 2011-2013 Source: PACT data Both the PACT and sub-partners’ teams identified the ISP as an important and useful document for ensuring action at the sub-partner level. However, there are some challenges:  Prioritizing issues between sections in the ISP is not documented; and  Prioritizing actions within an ISP section is also not identified. Without prioritization, an organization inevitably starts by addressing the easiest gaps. PACT’s FY2013 Annual Performance Review (APR) shows that many sub-partners have gaps relating to work that has to be completed by their boards, or gaps relating to the management of staff. These relate to the OD section and some selected gaps are captured in Table 7 from the ISPs of the 8 sampled organizations (organizations not identified to maintain confidentiality): Table 7. Selected OD gaps in sub-partners’ ISPs Organization OD Area Some examples of organizational gaps as reflected in ISPs for 2013/14 1 Governance and Leadership - Weak oversight mechanism of board (annual plans, budgets, regular monitoring of implementation and spending); - Lack of identification and selection procedures for board; (board recruitment procedures) Management Practices - Updating and consolidating human resource (HR) policies and procedures needed; - Lack of staff training and development plan - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 Governance and Leadership AVERAGE SCORES OCAT Assessment Results (PACT) Bantwana Cabrini Cheshire GSH Khulisa LDS Lutsango SCSWD SWABCHA SWANNEPHA TSA VOC AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 25 Organization OD Area Some examples of organizational gaps as reflected in ISPs for 2013/14 2 Governance and leadership - Board has not acted to ratify policies that have been developed by the partner - Governance training still to happen - Lack of resource mobilization plan - No finalized strategic plan Management Practices - Clarity of roles and responsibilities of board versus senior management is needed - Outdated organogram - Limited staff planning skills - Inconsistent annual appraisal of staff - HR policies and procedures not ratified and adopted 3 Governance and Leadership - Limited gender balance in board and staff complement Management Practices - Weak orientation of volunteers - Lack of staff training and development plan - Lack of budget for training plan 4 Governance and Leadership - Lack of clear separation of roles (board and staff) – board representatives not providing oversight, but supervision - Policies and procedures are reviewed without staff involvement - Board orientation not conducted due to delays in board buy-in on the activity - Lack of role clarity around board and senior management - roles and responsibilities - Lack of updated strategic plan Management Practices - Inconsistent use of staff appraisal systems - Inconsistent adherence to organizational policies and procedures - Limited planning skills in some areas in organization (context, program, budgeting, M&E, etc.) 5 Governance and Leadership - Advisory Board still not in place – further recruitment processes needed - Limited provision of progress update to the board by Executive Director (annual plans, budgets, regular monitoring of implementation and spending) - Lack of board manual - No recruitment procedures for selection and recruitment of board members Management Practices - Lack of consistent senior management meetings (weekly & documented) - Absence of performance management system - Lack of clarity on board’s role in mobilizing resources - Limited planning skills in certain areas 6 Governance and Leadership - Absence of resource mobilization plan - Exec. Director not appraised annually Management Practices - Lack of staff training and related development plan and budget - Absence of workplace HIV/AIDS program 7 Governance and Leadership - Lack of board manual to provide guidance on oversight functions - Lack of mapping of needs of beneficiaries Management Practices - Lack of budget for training of staff - Lack of clear management documentation, decisions, processes and activities. 8 Governance and Leadership - Executive Director not appraised annually - Absence of board charter - Lack of resource mobilization plan Management - Absence of updated organogram Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 26 Organization OD Area Some examples of organizational gaps as reflected in ISPs for 2013/14 Practices - Lack of budget for staff training - Lack of conflict of interest policy - Staff appraisal process not completed - Lack of clear policy of volunteers Source: PACT data The ISP is clearly set out and so is the corresponding action to close the gaps. Each sampled organization was asked to look at their latest ISP and to comment on what actions they had taken and what was difficult to change and what was not. There was a definite understanding of procedure, timing and responsibility levels from all the organizations. The gaps reflected in Table 7 ideally should have been dealt with in the first two years of PACT’s support, rather than waiting till the later stages of the project. This is because building the capacity of boards and putting organizational policies and procedures in place are the starting points in increasing the organizational capacity. In the other areas of capacity building, PACT has offered ongoing training supported by site visits where necessary. Monitoring, evaluation, reporting and learning (MERL) is an important area of organizational capacity building. It is where organizations learn how to collect, collate, verify and then analyze data in order to inform their practice, their donors and government on their activities and results. The MERL systems that have been put in place and supported by PACT are robust and effective. There has been a steady growth in organizational MERL proficiency among PACT sub-grantees (Figure 6), with the exception of one organization whose M&E officer resigned in 2012 which directly impacted on reporting to PACT. PACT provided extra support to the organization during that time and until a new staff member was employed and settled (see Table 8). The MCAT considers organizational functions such as accounting procedures, internal controls, financial management, and policy environment. Each of these sections is further divided into appropriate sub-sets which cover all aspects of accounting procedures and grant application. Figure 7 below records the MCAT assessment results from 2011 to 2013. Khulisa Umntfwana has made the most dramatic improvement in management capacity over the period of 2011 to 2013. This organization operated without any systems prior to becoming a PACT sub-grantee, with volunteers providing ad hoc services to the community on behalf of Swaziland’s Queen Mother. Their acceptance of the need for organizational systems, structures, and policies, has enabled it to grow significantly in ability and skills. On the other hand, SWANNEPHA showed a decline in progress, which was indicated in PACT’s FY2013 APR. SWANNEPHA’s performance was also captured through the ISP and reported as follows:  Grants and Finance: “The partner was changed from a direct grant to an in-kind after realization that proper accounting procedures, internal controls and financial management practices were not being implemented.”  MERL: “Provision of MERL technical assistance was constrained by the high turnover on the M&E officers.” Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 27 Figure 6. MERL assessment results for PACT sub-partners 2011-2013 (except for new sub-partners in 2013) Source: PACT data Figure 7. Overall MCAT assessment results for PACT sub-partners 2011-2013 (except for new sub-partners in 2013) Source: PACT data Figure 8. Accounting Procedures MCAT assessment results for PACT sub￾partners 2011-2013 - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 OVERALL AVERAGE SCORES MERL Assessment Results (PACT) Bantwana Cabrini Cheshire GSH Khulisa LDS Lutsango SCSWD SWABCHA SWANNEPHA TSA VOC AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 28 - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 Accounting Procedures AVERAGE SCORES MCAT Assessment Results (PACT) Bantwana Cabrini Cheshire GSH Khulisa LDS Lutsango SCSWD SWABCHA SWANNEPHA TSA VOC AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 29 Where there were problems, PACT worked directly with organizations to help through mentoring visits, although SWANNEPHA’s contract ended in September 2013 and was not extended further. To close the gaps recorded in the ISPs, support is given through formal training and mentoring visits. PACT’s support to each organization is given according to the need and request. Table 8 shows the number of organizational capacity-building mentoring visits and training workshops during 2012-2013 delivered by PACT to each sub-partner. Table 8. Organizational Capacity Building delivered by PACT 2012-2013 Sub-partner Mentoring Visits No. of workshops 2012/2013 Organizational Capacity Level MER 2012 Fin/Admin 2012 MER 2013 Grants 2013 OD 2012 OD 2013 Total per sub￾partner CANGO 4 4 27 11 3 26 75 7 OD Roadmap: 2.89 of 5 LDS 7 4 13 7 3 7 41 10 Marginal SWABCHA 5 2 10 5 4 8 34 7 Expanding SWANNEPHA 4 4 6 8 2 7 31 8 Marginal SCSWD 8 4 4 4 1 8 29 11 Expanding Cabrini 3 1 4 11 3 7 29 3 Expanding Khulisa Umntfwana 13 2 - 2 2 4 23 9 Marginal VOC 5 2 - 3 1 4 15 8 Marginal Bantwana 4 2 - 2 2 4 14 7 Expanding TOTAL 53 25 64 53 21 75 291 Source: PACT Technical Assistance Database Note: Some mentoring visits not captured in this table are those that relate to other support such as those to help leadership on specific needs. Attendance at the capacity-building workshops is by invitation, and the organizations that do attend identify a particular need for themselves, especially when there has been a change of staff. Capacity Building workshops generally last 1 to 4 days. Refresher courses are offered throughout program implementation in order to ensure that new staff are trained as soon as possible after they join an organization. Capacity building training workshops started in FY2011 with governance and leadership, which also was a focus in subsequent years. PACT’s capacity building also has a strong emphasis on finance training, grants management training, and MER training as shown in Table 9. PACT complements the delivery of formal workshops with ongoing, intensive mentoring of partners and sub-partners to provide more support and guidance. The visits are seen as crucial and essential by the sub-partners, who feel comfortable enough with PACT to phone and ask for direct help when needed. The average number of mentoring visits was 30 per organization in 2012-2013, ranging from 2 to 72 mentoring visits per organization per year. It could not be assessed whether the number of mentoring visits to all organizations was sufficient, however, according to the organizations: - PACT was always willing and available to them; and - PACT visited them regularly. Some organizations would like more visits as they feel their needs are not being completely met. This applies especially to CANGO, to which PACT emphasized mentoring of CANGO’s leadership. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 30 PACT’s country director worked one-on-one with the CANGO’s executive director on a regular basis. Although these support visits were above and beyond other mentoring visits, CANGO sees itself as needing more organizational and technical capacity for future role as a UGM and to support its sub-partners. In organizational capacity building, OD training (governance/boards, HR systems, vision and mission, etc.) is considered generally to be an overarching domain, or the starting point, under which all other organizational capacity building activities fit and follow-on. PACT’s OD support for organizations has focused on governance and leadership issues through strengthening NGO boards and setting in place HR policies and procedures. The training of boards started in August 2011, when 30 participants from Lutsango and Khulisa Umntfwana were trained on good governance and leadership. The next training was held in February 2012, when 23 participants from 12 organizations attended training for boards. The momentum for training boards picked up pace later in 2012 and 2013, as shown in Table 9. It is a concern that training of boards and setting up of HR procedures was not a focus at the beginning of the REACH project. Capacity building in an organization usually starts with training the board and strengthening internal organizational leadership. A common starting point is refining an organization’s vision and mission, followed by strategic planning. At the same time, training on governance should begin unfolding, so the board and organizational leadership know their roles and responsibilities and accept accountability for them. There is presently a backlog of OD work that has to be completed, mainly with organizations that PACT considers not fully mature. Organizational sustainability depends on strong leadership and governance, which needs to be firmly in place by the end of the REACH project. There is a question as to whether this is possible in the time remaining in the project. According to one sub-partner executive director, “To be honest with you, I think PACT’s strongest point is, in my view at least (I have been working with them), their OD support, which I think has been a huge impact, where they are looking at issues around governance, role of the board, management tools, finance, program management and monitoring and evaluation.” The success of PACT’s organizational capacity building approach was captured also in the online survey. The majority of respondents who provided comments (72% of 22) were positive about PACT’s capacity building approach, as paraphrased in Table 10 below. Table 9. Organizational Capacity-building Workshops – FY2011-2013 Workshop Date Workshop Topic No. of Partners Sampled sub-partners that attended August 2011 Governance and Leadership 2 Khulisa Umntfwana November 2011 Grants Management 14 Bantwana, Cabrini, CANGO, Khulisa Umntfwana, LDS, SCSWD, SWABCHA, VOC December 2011 Financial Management 14 Bantwana, Cabrini, CANGO, Khulisa Umntfwana, LDS, SCSWD, SWABCHA, SWANNEPHA February 2012 Governance and Leadership 12 Bantwana, Cabrini, CANGO, LDS, SCSWD, SWABCHA, SWANNEPHA, VOC February 2012 Basic Monitoring and Evaluation 13 Bantwana, Khulisa Umntfwana, LDS, SCSWD, SWABCHA, SWANNEPHA, VOC May 2012 Resource Mobilization 12 Bantwana, CANGO, Khulisa Umntfwana, LDS, SCSWD, SWANNEPHA, TSA, VOC May/June 2012 Data Quality Management 14 Bantwana, Cabrini, Khulisa Umntfwana, Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 31 Workshop Date Workshop Topic No. of Partners Sampled sub-partners that attended LDS, SCSWD, SWABCHA, SWANNEPHA, VOC July 2012 Volunteer Management 14 Bantwana, Cabrini, CANGO, Khulisa Umntfwana, LDS, SCSWD, SWABCHA, SWANNEPHA, VOC August 2012 Basic Excel and Data Analysis 14 Bantwana, CANGO, Khulisa Umntfwana, LDS, SCSWD, SWABCHA, SWANNEPHA, VOC November 2012 Strategic Communications Training 10 CANGO, Khulisa Umntfwana, LDS, SCSWD, SWANNEPHA, VOC November 2012 Data Quality Management Training 7 Bantwana, Khulisa Umntfwana, SCSWD, SWABCHA, VOC. November 2012 Board Training for SCSWD 1 SCSWD December 2012 Strategic Planning Meeting for Cheshire Homes 1 Cheshire Homes December 2012 Board Training for VOC 1 VOC February 2013 Board Training for Khulisa Umntfwana 1 Khulisa Umntfwana March 2013 HR Training for Lustango 1 Lustango March 2013 HR Training for SWANNEPHA 1 SWANNEPHA March 2013 Board Training for Cheshire Homes 1 Cheshire Homes March 2013 HR Training for Bantwana 1 Bantwana May 2013 Board Training for CANG0 1 CANGO July 2013 Succession Planning Training 1 SWABCHA September 2013 HR Training for SCSWD 1 SCSWD October 2013 Grants Training 8 CANGO, LDS November 2013 Board Training for LDS 1 LDS December 2013 Basic MER Training 8 CANGO, SCSWD December 2013 Finance Training 8 CANGO, Khulisa Umntfwana, LDS Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 32 Table 10. Online survey respondents’ opinions of PACT’s organizational capacity building (N=22) Positive opinions Negative opinions  Well-conceived and well-run workshops, with helpful exercises;  Regular mentoring support after workshops;  Monitoring is constant and highly appreciated;  Immediate response to queries after training;  Best capacity building organization in Swaziland;  Professional approach by PACT staff that helps organizational, professional growth;  Capacity building is good and there is a flexible approach to dealing with organizational needs;  Individual skills as well as organizational skills have been improved;  Skilled individuals at PACT;  Capacity building is a step by step approach to community development;  M&E is excellent at all levels;  M&E individual skills have improved;  The skills in M&E help to measure impact;  CANGO has started to be capacitated;  Highly appreciative of the capacity building approach and model;  The approach is most appreciated and welcomed in the organization.  Sometimes the training manuals do not match the presentations;  CANGO is not yet effective enough to deal with all capacity building needs;  Need time frames for capacity building to know how and when we will be able to ‘go it on our own;’  Capacity building is focused on PACT activities and should be on the organization as a whole;  Need to expand the OD staff to include more experienced individuals who can interact effectively with highly experienced board members;  Differing skills levels among PACT staff affect capacity building;  More team building of sub-partner staff within organizations is needed. These opinions reflect the success factors of PACT’s organizational capacity building approach and also some areas that need improvements. One quote from the online survey sums up the general feeling about PACT quite succinctly: “I have worked with a lot of development organizations in Africa and around the world, and I consistently think of PACT when describing an ideal model of capacity building and support. Our organization had strong implementation abilities before our partnership with PACT, but PACT has made them better, and more importantly has drastically improved our support systems so that implementation can continue to happen effectively and efficiently. They are excellent at knowing which issues need to be addressed first and which can wait. The result is a consistent trajectory towards overall organizational improvement.” To reinforce these opinions, information gleaned from the KIIs showed that sub-partners view the OCAT tool positively, including the ISP. There also is a high level of trust between the PACT employees and the sub-partners because of the way PACT includes all staff members in the rating of organizational effectiveness. This level of trust is an essential part of the capacity-building approach of PACT and is the key foundation from which interventions are able to be introduced. Inevitably, there are some weaknesses, and in this case, issues around the scheduling of workshops or visits that were noted by several key informants. “It was somehow a disadvantage to me. You see some of the trainings…or the meetings that they call…usually towards the end of the month…clash….with our deadlines. At the end of the month, that’s the time when we are expected to be in the office all the time.” Overall, however, PACT is seen as providing a good capacity-building service. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 33 “I feel like they’ve really done a great job in capacity building. I would hate to lose PACT…I think we still have a lot more capacitation that we need.” (OVC Manager) “But the strength has been at mentoring, following up now what has happened at the training and the one-on-one partner visits, both with CANGO and with the partners. I feel like that is what sets them apart, because the training is just unveiling the topic and then the one-on-one, you are now looking at the outcomes of the training.” (Project Manager) The combination of training and mentoring, along with using the ISP as the instrument against which action is debated and confirmed, is seen as the essential aspect of PACT’s organizational capacity building model. Capacity Building of CANGO PACT support to CANGO must fulfil two capacity-building objectives. The first one is to increase the capacity of CANGO to run as an effective organization through developing its internal systems and staff; the second is to ensure CANGO can operate as an UGM and oversee the work of sub￾partners. According to a key informant at CANGO: “I think it’s been great, impact is great, because…there were no M&E systems for CANGO before this project, even though CANGO had been doing some grants managements before…there were no M&E systems either way...but with the assistance from PACT… now put in place M&E systems for the grants management unit and for CANGO as an organization.” The more established sub-partners believe that CANGO is slow in responding, and consistently checks back with PACT before it takes action, although it needs more training. Even though many see a positive change in CANGO, which is now better skilled and better able to cope with the stresses of managing sub-partners, there is still a belief that CANGO needs further capacity building before it can become a competent UGM. More detail on CANGO’s progress toward becoming a UGM is presented in the answer to Question 7. CONCLUSIONS  PACT provides a good, well-considered and appropriate range of capacity-building services through training and mentoring to sub-partners in Swaziland, but there are some areas that could be improved;  The introduction of OD leadership and governance training and related activities, which has been well-received, has occurred quite late in the project’s life span and should have been introduced earlier;  The capacity-building process is well-received where the leadership of the organization is strong;  In terms of training and capacitating CANGO to be a UGM, there are mixed reviews from the CANGO sub-partners on CANGO’s capability. Some sub-partners perceive CANGO to still be a learning organization that is delivering a capacity-building service which needs further strengthening. RECOMMENDATIONS  PACT should continue OD capacity building by working with all organizational boards and Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 34 senior management (including CANGO) to further develop understanding about governance, roles and responsibilities, management and leadership, and at the same time assist the organizations to finalize succession plans as they develop and refine their organizational charts;  PACT should finalize its HR capacity building with all the sub-partners;  PACT should expand mentoring support visits to all organizations where there is a need, as these are well-received and were mentioned by all the sub-partners as an essential part of capacity building;  Capacity building in OD should be started at the start of any similar program;  PACT should ensure that all ISP actions are prioritized by sub-partners through setting in place a prioritized action plan that is followed; and  PACT should run its training workshops at times when there is no clash with end-of-month reporting duties. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 35 Question 2. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? FINDINGS PACT provides technical capacity building for its sub-partners in OVC, HIV prevention and gender. At the direction of PEPFAR, PACT did not have a strong focus on technical capacity building on HIV treatment and support to the two sub-partners implementing care and treatment services, because another specialized PEPFAR partner was assigned to provide technical capacity building in this area. Pact’s technical support is provided through workshops, technical working group meetings, and individual program and technical implementation site visits. For example in FY2012, 89 program site visits focused on targeting, implementation planning and budgeting, and technical site visits were made by PACT to 12 sub-partners implementing services. In FY2013, 105 program and technical site visits were made by PACT to 7 sub-partners implementing services (not including CANGO). However, several sub-partner key informants noted that the quality and usefulness of the technical capacity building site visits varied based on the skill set of the provider. The topics of PACT’s technical capacity building workshops for more than one organization included HIV prevention (1 workshop), gender (3 workshops), WORTH (women’s empowerment programming) (3 workshops), OVC quality standards (2 workshops), child protection (1 workshop) and psycho-social support training (1 workshop). The relevant organizations appear to have attended the workshops held by PACT in their programmatic areas, except gender design, which arguably could have included all the sub-partners as gender is a cross-cutting HIV/AIDS technical and programmatic issue. Based on the direction from PEPFAR, PACT has not provided training to any sub-partners on HIV/AIDS treatment/care, although two treatment partners have received treatment support from Columbia University’s International Center for Care and Treatment Programs (ICAP) at PEPFAR’s direction. Table 11 presents PACT’s sub-partner technical capacity-building workshops conducted from November 2010 through December 2013. Table 11. Technical Capacity Building Workshops conducted by PACT (FY2011-FY2013) Workshop Date Workshop Focus No. of NGOs Participating November 2010 Gender Design 3 January 2011 OVC Quality Standards 10 February 2011 WORTH Orientation 2 February 2011 HIV Prevention NATICC only March 2011 Gender Mainstreaming 10 July 2011 WORTH Empowerment Worker and Management Committee Training 2 December 2011 Child Protection Cabrini Ministries only January 2012 Gender Mainstreaming 15 April 2012 Child Protection 12 November 2012 Psycho-Social Support (PSS) 10 December 2012 Roll-out OVC Quality Service Standards 32 January 2013 HIV Prevention 8 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 36 Workshop Date Workshop Focus No. of NGOs Participating February 2013 Child Protection Salvation Army only April 2013 Child Protection Cheshire Homes only June 2013 Child Protection LDS only June 2013 Child Protection SCSWD only December 2013 WORTH Business and Marketing Training 2 While 11 technical capacity building workshops have been conducted by PACT over the past 3 years, the range of the HIV/AIDS technical services workshops has not been wide. Some workshops were requested by sub-partners, such as assistance on economic strengthening activities and on child protection, as noted by key informants. Several sub-partner key informants also requested more frequent technical capacity-building workshops due to staffing changes and to keep up-to-date with effective HIV prevention intervention approaches. Some key informants noted the workshops were of higher quality when PACT included outside experts as facilitators. Balancing the time availability of sub-partners for technical HIV/AIDS workshops with organizational capacity-building workshops was the main constraint PACT noted for not providing more HIV/AIDS technical workshops. One key informant told the team, “I think with the PACT support on the projects, the range of services provided are actually not as wide as what we provide as an institution.” Nonetheless, PACT’s technical capacity building was appreciated by about 75% of its sub￾partners according to the anonymous online survey results (Figure 9). About 70% of the respondents stated that PACT offered additional support after conducting technical capacity building workshops to their organizations. Moreover, most sub-partner responses in the online survey attributed their better service quality to PACT’s technical capacity building. The evaluation team administered PACT’s self-assessment Technical Capacity Assessment Tool separately with the sample of 8 sub-partners who deliver HIV/AIDS services. The 2014 technical capacity assessment scores were plotted against the sub-partners’ previous assessment scores in 2011, 2012, and 2013 (see Figure 10 through Figure 13). Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 37 Figure 9. Effectiveness of PACT’s technical HIV/AIDS capacity building of sub-partners The sub-partners who showed the smallest increase in technical capacity assessment scores from 2011-2014 were the two that implement treatment and care services (Cabrini and SWANNEPHA). Cabrini started out with the highest technical capacity assessment score of any of the sampled sub￾partners (4.322). Thus Cabrini already provided high-quality services when it first started receiving support from PACT in 2011. Nevertheless, Cabrini showed a slight decline in 2014 from their score of 4.76 in 2013, albeit the highest technical capacity assessment score of any sub-partner at that time. The decline in their score may be due to the team adding HIV prevention in 2014 to the technical areas on which Cabrini scored themselves in 2013, the national stock-out of HIV testing reagents in January 2014 affecting their HIV counseling and testing services, and administration of the assessment tool by the team rather than PACT. On the other hand, SWANNEPHA’s decline in their in 2014 score from 2013 was due to the end of their PACT funding in September 2013, which has had a negative impact on their service delivery capacity in 2014. 2 On a scale of 1 to 5 in 2011, which means “needs some minor adjustment but without urgency” 0.0% 25.0% 50.0% 75.0% 100.0% strongly disagree disagree neutral agree strongly agree PACT is highly effective in building technical HIV/AIDS capacity in my organisation (N=34) S Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 38 Figure 10. Overall technical assessment results 8 sub-partners 2011-2014 Data sources: 2011-2013 from PACT; 2014 from evaluation team As an example of PACT’s progress on technical capacity building: in FY2013, 42% of PACT’s sub￾partners (5 of 12) improved by at least 30% in their relevant technical capacity area, although PACT’s target for technical capacity improvement was 75% of the sub-partners improving by 30% or reaching technical capacity level 4. Moreover, 1 sub-partner showed no improvement in their technical capacity in FY2013 while another showed a slight decrease in their technical capacity. It thus appears that there is a continuing need for technical HIV/AIDS capacity building of most of PACT’s sub-partners. Khulisa Umntfwana showed the largest increase in their technical capacity scores between 2011 and 2014 – from 2.60, “needs substantial attention,” to 3.86, “needs some improvements.” Lutheran Development Service (LDS) also showed a substantial improvement in their technical capacity assessment scores between 2011 and 2014, rising from 2.60 to 4.04, “needs some minor adjustment but without urgency.” These significant increases in technical capacity scores emphasize that PACT’s technical capacity-building workshops had positive effects on the technical capacities of the two sub￾partners: on HIV prevention services at Khulisa Umntfwana and on OVC services at LDS. It is important to note that more than 60% of sub-partner respondents to the online survey reported that they also received technical capacity building from other organizations. Nearly 50% of those respondents reported that the quality of the technical capacity building they received from other organizations or individuals was about the same as that provided by PACT. All the sub- - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 2014 OVERALL AVERAGE SCORES Technical Assessment Results (PACT) Bantwana Cabrini Khulisa LDS SCSWD SWABCHA SWANNEPHA VOC AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 39 partners agreed that PACT had built their capacities organizationally, but not all agreed that PACT had built their technical capacities. Moreover, all the sub-partners expressed a desire for additional HIV/AIDS technical capacity building. This need was mentioned by many key informants, including the need for refresher technical HIV/AIDS training by some on prevention and on treatment and care. Technical Capacity Building in SEXUAL PREVENTION Figure 11 below documents the technical capacity assessment scores between 2011 and 2014 for 3 sampled sub-partners implementing sexual prevention services – Khulisa Umntfwana, SWABCHA, and VOC – and the 2014 score for Cabrini. The three sub-partners scored higher in HIV prevention each year after they started receiving technical capacity building by PACT, with the technical capacity assessments administered by the evaluation team in 2014 showing the highest scores. The sampled sub-partners have accrued higher scores in HIV sexual prevention over time. PACT held one capacity building workshop for its sub-partners in HIV prevention over 3 years (as per Table 11). However, a few sub-partners received substantial technical assistance on prevention, and the capacity building workshops on gender related to HIV prevention were additional assets. PACT also formed a prevention technical working group (TWG) for its sub-partners providing HIV prevention services. However, PACT sub-partners operate in only one or two regions in Swaziland, similar to most civil society service providers, and a government official noted the limited scale as the biggest concern: “There is a consistency in the delivery of approaches, yet an inability to get the overall scale to turn the tide in prevention…the scale needed geographically, or greater coverage.” Technical Capacity Building in HIV TREATMENT AND CARE The evaluation team administered PACT’s technical capacity assessments of treatment at 2 sampled sub-partners implementing HIV treatment/care services from 2011 to 2014 – Cabrini and SWANNEPHA (see Figure 12). Cabrini showed a slight increase in its already very high technical capacity assessment score in treatment between 2011 and 2014. However, the rise in its score cannot be attributed to PACT’s technical capacity building as PACT has not built Cabrini’s capacity in treatment and care services, although PACT has provided some programmatic technical assistance for these services. Columbia University’s International Center for Care and Treatment Programs (ICAP) provides backstopping support for Cabrini’s treatment and care services through weekly visits. On the other hand, SWANNEPHA showed a lower treatment assessment score in 2014 compared to 2013 due to PACT ending its funding support to the sub-partner in September 2013, prompting about a 50% cut in its treatment-related services provided by ‘expert clients’. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 40 Figure 11. PREVENTION Technical Assessment results 3 partners 2011- 2014 and 1 in 2014 Data sources: 2011-2013 from PACT; 2014 from evaluation team Figure 12. TREATMENT and CARE technical assessment results 2 partners 2011-2014 Data sources: 2011-2013 from PACT; 2014 from evaluation team - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 2014 Treatment AVERAGE SCORES Technical Assessment Results (PACT) Cabrini SWANNEPHA AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 41 Technical Capacity Building in OVC The evaluation team also administered PACT’s technical capacity assessments with the four organizations implementing OVC services and plotted the 2014 scores against the scores for the sub-partners in 2011-2013, except Bantwana’s OVC assessments, which started in 2012. Figure 13 shows a rise in all sub-partner’s scores except for Cabrini, which showed a slight decrease in 2014, perhaps attributable to more circumspection from the larger group at Cabrini who contributed to the self-assessment of their OVC services in 2014 than in 2013, when they rated themselves as needing no improvements. In 2014 the Cabrini group agreed that their range of OVC services focuses primarily on students and does not reach every household. Figure 13. OVC technical capacity assessment results 4 partners 2011-2014 Data sources: 2011-2013 from PACT; 2014 from evaluation team. PACT’s capacity building on OVC services was lauded by the majority of key informants interviewed. One sub-partner commented about PACT’s OVC work, “Last year there was the OVC quality standard…PACT was very much involved in that and they helped us as a country…to come up with something that will help us track and improve the delivery of quality services.” - 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 2011 2012 2013 2014 OVC AVERAGE SCORES Technical Assessment Results (PACT) Bantwana Cabrini LDS SCSWD AVERAGE Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 42 CONCLUSIONS  Technical capacity assessment scores in prevention and OVC of sampled sub-partners show overall increases.  Over November 2010 to December 2013, PACT conducted only 11 technical capacity building workshops for more than 1 sub-partner organization, for an average of three technical workshops per year – not a large number. Key informants identified a need for more HIV/AIDS technical workshops encompassing HIV prevention and treatment and care interventions for service providers.  PACT’s focus on building capacity on children-focused services, including OVCs, and gender￾focused service delivery, is a strength in its technical capacity building and, according to most of the relevant sub-partners, has raised the quality of their HIV prevention and OVC services.  PACT’s technical capacity assessment tool involves annual self-assessments by these health and support service providers to gauge their progress on increasing their capacity to deliver high-quality services. Whether annual self-assessment is an adequate approach for gauging the quality of an organization’s technical service delivery is questionable, especially when the technical capacity building provided to the sub-partners has been limited. RECOMMENDATIONS  PACT should consider more frequent technical capacity building workshop, but retain an appropriate balance between workshops and technical field support to sub-partners.  To help scale up Swaziland’s national HIV/AIDS response in the future, technical capacity￾building workshops by PACT could include a larger number of organizations beyond sub￾partners, such as NERCHA and other partners, although additional funding likely would be needed.  External assessments of technical HIV/AIDS service delivery by technical experts to augment organizational self-assessments would provide a more objective gauge of the quality of service delivery and additional technical HIV/AIDS capacity-building needs. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 43 Question 3. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? FINDINGS Enhanced Quality of Services Delivery Most sub-partner respondents to the online survey felt that PACT’s technical capacity building has helped them deliver higher quality services (Figure 14). More than 90% of respondents believe that they are delivering a greater range of high quality services as a result of PACT’s capacity building. One sub-partner key informant believed that PACT’s capacity building has helped the organization to function better and to raise the quality of their HIV/AIDS service delivery, even though PACT has not built their capacity across the full range of HIV services that they provide. Figure 14. Increased range of sub-partners’ high-quality services based on PACT’s capacity building PACT’s assistance to the sub-partners focusing on the quality of HIV/AIDS services includes alignment with PEPFAR guidelines and indicators and alignment with Swaziland’s National Multi￾sectoral Strategic Framework for HIV and AIDS. A sub-partner key informant noted about service quality: “We have developed so much through PACT’s capacity building as individuals. We have acquired skills that are helping us in ensuring that we deliver quality services, and we are always on our way to reaching top quality.” 0.0% 25.0% 50.0% 75.0% 100.0% strongly disagree disagree neutral agree strongly agree PACT technical HIV/AIDS capacity building helps our organisation deliver a greater range of high quality services to our beneficiaries (N=35) S Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 44 A number of government key informants felt that PACT’s assistance to the sub-partners had raised the quality of the services being delivered and the targeting of these services. According to one government informant: “The mere fact that they are drawing their program and support from the national strategic framework shows alignment, and having their own targets, their implementers support the national framework.” Expanded Numbers of Persons Reached/Targeted Sub-partners report receiving support from PACT for better targeting of beneficiaries. More than 85% of online survey respondents agreed that PACT’s assistance has helped them expand the number of beneficiaries they target (see Figure 15). When supporting sub-partners in the area of targeting, PACT reported that it always looks at sub-partner capacity and budgets, and then works with sub-partners to make sure that their capacity is aligned to what they can cover. The size of the budget for service delivery obviously affects targeting calculations. One sub-partner key informant noted, “We started as low as three thousand, but now we are up to eighteen, twenty thousand and in fact, we are going to twenty-four thousand.” Figure 15. Expansion of beneficiary numbers based on PACT’s assistance Figure 16 shows that more than 90% of the respondents to the online survey agreed that PACT’s assistance also has helped their organizations reach the appropriate beneficiaries. To verify these online survey responses, we examined the annual targets and results for FY2011- FY2013 for the interventions implemented by the 8 sampled sub-partners to confirm that both the targets and results had indeed increased. Several examples below demonstrate that sub-partner’s reach has increased, although it is difficult to attribute these increases directly to the PACT support provided to each sub-partner. 0.0% 25.0% 50.0% 75.0% 100.0% strongly disagree disagree neutral agree strongly agree PACT’s assistance has helped us to expand the number of beneficiaries that we support (N=35) S Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 45 Figure 16. Appropriateness of Targeting based on PACT’s assistance Source: Online Survey One example is presented in Table 12 below, which shows an increase from 2011-2013 in most sub￾partners’ targets, actual reach, and overall results for Umbrella Care, the PEPFAR indicator for the number of adults and children provided with a minimum of one care service. While sub-grantee performance was mixed in FY2011 and FY2013, all partners exceeded their targets in FY2012. Overall, the actual numbers reached exceeded targets in FY2012 and FY2013. Table 12. Umbrella Care Targets, Actuals, and Results for 5 sub-partners 2011-20133 Source: PACT data Another example is from PACT’s 2013 Annual Progress Report (APR), where 6 of 13 sub-partners (46%) delivering services in 2013 reached 85% of their annual targets for 75% of their indicators. Table 13 shows that as a group, the sub-partners met 7 of the 9 technical area targets by more than 100%. Thus PACT’s support to its sub-partners around targeting appears to have had a very positive effect on sub-partners’ reaching most of their targets in FY 2013 3 The sparklines in the RESULT column show if the actual numbers reached were greater or less than the target. 0.0% 25.0% 50.0% 75.0% 100.0% strongly disagree disagree neutral agree strongly agree PACT ‘s support helps us to reach the appropriate beneficiaries (N=35) S Target Actual Result Target Actual Result Target Actual Result Bantwana 750 846 7 332 5 200 15 826 8 024 Cabrini - 1 837 2 778 1 800 2 323 2 500 LDS - - 6 205 6 020 8 507 9 426 SCSWD 3 500 7 016 17 500 23 007 18 587 23 083 SWANNEPHA 20 160 6 846 11 000 16 724 1 779 4 230 TOTAL 24 410 16 545 41 520 56 046 42 767 51 518 Umbrella Care - No. eligible adults/children provided with a minimum of 1 care service Organization 2011 2012 2013 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 46 Table 13. FY2013 targets and results for interventions implemented by 13 PACT sub-partners Intervention Area Target Actual Reached % of Target ART Services 3,390 7,822 231% Support Care 40,849 54,733 134% Umbrella Care 48,268 63,873 132% Clinical Care - Overall 11,019 13,729 125% Gender: GBV 20,415 24,654 121% Gender: Male Norms 11,350 13,133 116% Sexual Prevention 12,390 13,787 111% HTC 6,079 5,403 89% Gender: Income and Productive Resources 2,187 1,200 55% Source: PACT data A third example comes from the OVC Focus Groups, where most stakeholders and beneficiaries (i.e. the caregivers, parents/guardians and OVCs themselves) believed that the relevant beneficiaries are being targeted in communities. Figure 17 shows that more than 90% of respondents believe that the targeting of OVC programs was excellent (75%) or good (17%), and more than 85% of respondents believe that the targeting of relevant children for OVC services was excellent (58%) or good (27%). Caregivers and beneficiaries, including the OVCs and parents/guardians, who participated in FGDs agreed that sub-partner projects have made a significant improvement in the lives of many OVC. “We were abandoned before. We would need a hundred mouths to express our thanks.” Although FGD respondents were largely satisfied with the targeting of OVC services, they were less satisfied with the geographic coverage, which was acknowledged to be due to resource constraints, rather than the design and implementation of the programs per se. One explanation is the remote location of some communities, which has resulted in OVC in these communities receiving few or no services. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 47 Figure 17. OVC targeting and reach by goods and services according to OVC, parents/guardians, and community caregivers Source: OVC scorecard data CONCLUSIONS  According to the responses of most key informants, the quality of the services provided by the sub-partners has improved based on PACT’s support.  Both program staff and beneficiaries alike report that the targeting of services delivered has improved through PACT’s support. The results of some services delivered by the sub￾partners increased substantially in 2012 and 2013, with care and treatment services showing the greatest increases in results.  The coverage of some services has increased based on PACT’s support to its sub-partners through better and expanded targeting. In the OVC area, wider service coverage was acknowledged by beneficiaries, caregivers and parents/guardians to be constrained due to resource constraints, rather than the design and implementation of the programs per se. RECOMMENDATIONS  PACT should continue emphasizing quality service provision by sub-partners as the need for the delivery of high-quality HIV/AIDS services in Swaziland continues. Likewise, PACT’s emphasis on strengthening sub-partners’ service dosage and frequency should be maintained.  In its next round of funding, USAID should examine whether the levels of funding for staffing and service delivery are adequate for sub-partners capable of managing funds efficiently and effectively, in order to be able to reach more people in Swaziland and provide wider access to HIV/AIDS services. 76% 17% 2% 3% 3% 58% 27% 9% 4% 2% 0% 25% 50% 75% 100% Very Poor Poor Average Good Excellent Community Perspectives on OVC Targeting (N=230) Relevant children in community targeted by project Relevant children reached by goods and services Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 48 Question 4. How does different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? FINDINGS ON GEOGRAPHIC COVERAGE AND QUALITY AND DOSAGE NGOs working in the area of HIV AIDS are present in all four regions of Swaziland – Lubombo, Manzini, Hhohho, and Shiselweni – but most work in Lubombo because it has the highest prevalence of HIV/AIDS in the country. The socio-economic conditions of the region are very poor, and there are many OVC and associated challenges in the area. The services offered by the sampled sub-partners, and their reach figures for 2012 and 2013, along with the 2013 funding levels, are shown in Table 14. Table 14. Sampled sub-partners’ range of services, regional coverage, reach and funding Sub-partner Technical Area Regional coverage Reach 2012 Reach 2013 2013 PACT Funding Level (USD) Cabrini Ministries Treatment, Care and Support, OVC, Prevention, HCT, TB/HIV Lubombo 2,778 2,789 $314,419 Save the Children Swaziland (SCSWD) OVC, Gender Hhohho, Manzini, Lubombo, Shiselweni 23,007 23,083 $293,750 Bantwana Initiative OVC, Gender Lubombo 7,128 15,826 $270,750 Lutheran Development Service (LDS) OVC Lubombo, Shiselweni 6,271 8,507 $270,750 Swaziland Business Coalition Against HIV & AIDS (SWABCHA) Prevention, Gender, HCT Hhohho, Manzini, Lubombo, Shiselweni 7,160 8,853 $152,405 Voice of the Church (VOC) Prevention AB, Gender Hhohho, Manzini, Lubombo, Shiselweni 7,800 7,015 $121,116 Swaziland National Network of People Living with HIV & AIDS (SWANNEPHA) Treatment Hhohho, Manzini, Lubombo, Shiselweni 15,320 1,799 $97,451 Khulisa Umntfwana Gender, Sexual Prevention Hhohho 2,118 2,335 $60,000 Each sub-partner’s geographic coverage is dictated by the availability of funding, the type of intervention, and the ability to find qualified staff. Where an organization relies on volunteers, there generally is a supply of community workers available to support an intervention. However, volunteers tend to stay longer with an organization when they receive stipends. Skilled personnel are more difficult to find and to retain, especially in the rural areas. These factors can influence the ability of the sub-partner to extend its services geographically. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 49 Because there is a difference in geographical reach and in the number of communities served4, it is impossible to determine whether an increase in funding expands reach, as there is no apparent correlation between the funding received, the reach, and the number of sites. Also, several sub￾grantees receive funding from other donors which support their services in the same geographic areas as the USAID support from PACT. For purposes of reporting to PEPFAR, PACT has defined dosage according to the intervention. For example, under gender and prevention, sub-partners use “3 contacts” for counting one beneficiary reached. For other interventions, a beneficiary can be counted if the sub-partner only has one contact with them. In addition, dosage varies by sub-partner. For example, LDS visit NCPs weekly to support them, whereas SCSWD’s PSS volunteers do not see each child weekly. Program design, as well as the funding level and availability staff, affect dosage, with each project being handled individually. To see whether geographic coverage has an impact on dosage, we compare the work of one organization that works extensively in the Lubombo Region (Cabrini Ministries) to that of SCSWD that works nationally. The focus of the comparison was on services provided to OVC to answer the question: Is dosage different for those sub-partners that have a bigger geographical coverage? Cabrini Ministries Cabrini Ministries has worked in Swaziland for over 40 years, mainly in the Lubombo area, recognizing the needs of the deeply impoverished region. Cabrini does not want to expand its services outside Lubombo where it is known and well-accepted. Cabrini recognizes that geographic expansion could dilute the quality of services it offers as it does not have the necessary financial or human resources for expansion, especially the HR. Cabrini’s coverage is specific and focused, employing local Lubombo staff where possible with non-local staff members housed at the St. Philip’s Mission. Cabrini has diverse funders which can help sustain it after REACH ends. Their work in the area has deepened and expanded, which has ensured trust and acceptance by the community: According to a key informant commenting on communities in Lubombo, “They’re very wary of anybody that’s an outsider, and I don’t just mean national outsider, I mean other Swazis. It’s a pretty closed community.” Save the Children Swaziland (SCSWD) In contrast, SCSWD has a nationwide footprint. It is a member of Save the Children International and through this worldwide network has layers of support. Within Swaziland, SCSWD has six regional offices. The head office is in Mbabane, the capital, and quality control is directed from there to the regional offices. Within each region, the beneficiaries have access to local staff and localized services. However, to oversee and manage the quality of service delivery across all regions is reportedly a challenge. 4 For example, SCSWD services 4 districts, and 11 communities from within these districts, using 19 sites or office bases. Cabrini services 1 district and 5 communities across the district, with 25 sites of activity, including clinics. On the other hand, Bantwana works in 34 schools which encompass 27 different communities around the schools. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 50 Table 15 shows that SCSWD’s national footprint allows it to reach more than 15 times more OVC than Cabrini (per APRS for 2013 and 2013). Even though Cabrini’s OVC reach is lower, the range of services it provides to OVC is greater than those of SCSWD and their service model is more intense. It includes providing food, shelter, health care, education, legal services/child protection, PSS and economic strengthening. SCSWD largely concentrates on targeting more individuals for PSS, legal services/child protection and economic strengthening, and does not provide food and shelter or the other services provided by Cabrini. This is not because of geographic difference and distance, but because of the different service model used by the organization. Table 15. OVC Reached by Cabrini and SCSWD (2012-2013) Organization 2012 2013 Cabrini 914 1,056 SCSWD 17,500 20,286 Source: PACT data The comparison above demonstrates the difficulty in determining whether geographical coverage makes a difference in dosage when the sub-partners provides different services. However, we may surmise that concentrating services in one geographic area with the appropriate level of funding, leads to beneficiaries being accessed more frequently by staff who need not travel extensively. Factors affecting quality and dosage of services The main factors affecting service quality and dosage, whether in one region or more, are considered to be the quality and attrition of personnel. Coverage and quality of services depends on having the right number of support people living in or being able to travel to the area being served. As such, staff attrition is a major concern for all PACT sub-partners: for capacity building, for sustainability, and for geographical coverage. As stated by a key informant: “You build the capacity of a program manager and you think, oh, we are getting somewhere. Before you know it, they move on. You start again, so…staff attrition has been one…major challenge.” Service quality can also be affected by standards, as was shown in the piloting of the OVC QSS, which was conducted to determine if implementing standards improves the quality of programming and service delivery. The PACT report (2012) states, “a comparison of baseline and endline indicators shows there was an improvement in 20 out of 27 indicators (representing 74%) that were being tracked. The percent of indicators which improved by at least 5% during the pilot was 37%. Twenty-six percent of the indicators improved by at least 10%”. Establishing the QSS, which are now accepted guidelines of service delivery, gave Swaziland OVC service providers a common set of quality statements to work towards and to achieve. The OVC QSS set down what an organization should aim towards to provide excellent OVC service. Without a baseline assessment on quality service delivery, a sub-partner usually has no idea which way to work and what to aim for; but the QSS has helped to ensure service delivery for OVC is understood and standardized, which leads to quality service delivery. The QSS are very concise, easy to understand, and inherently serve as a capacity assessment tool that sub-partners can use to internally assess their own capacity in delivering quality OVC services. The QSS also helps sub-partners to identify which indicators to track and the types of activities to achieve stated outcomes. By using the QSS, each organization is able to collect baseline data on the beneficiaries to document progress and improvements in service delivery and OVC outcomes. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 51 Other standards have been developed including those ECCD, which were initiated by the MoET to which PACT provided inputs. They are presently being rolled out across Swaziland. Respondents’ views on geographic coverage quality and dosage of services Figure 18 shows that more than 75% of respondents to the online survey agreed that the dosage of their services to their beneficiaries has increased based on PACT’s monitoring and mentoring. While dosage data was not available for review, discussions about PACT’s assistance with dosage were held with key informants. One sub-partner key informant noted a discussion she had with PACT around dosage and the importance of increasing and strengthening the dosage of services: “I think in terms of service delivery they want to be sure on the issue of the dosage that if you say you have reached a person, really what have you said to that person can make that person be informed enough to say, ’Wow, this is risky behavior. Let me change my behavior because I’ll be putting myself at risk if I continue doing this!’” Figure 18. Increased frequency of beneficiary support based on PACT’s assistance FINDINGS ON SUB-PARTNERS ENGAGEMENT AND ALIGNMENT WITH NATIONAL STRATEGIES AND SYSTEMS, AND REFERRALS TO GOVERNMENT SERVICES: Several strategies are underway by the Government of Swaziland to improve the response to the HIV/AIDS epidemic. These include, amongst others:  Development and finalization of the extended National Multi-sectoral Strategic Framework on HIV and AIDS 2014-2018 (including prevention; treatment, care and support; and impact mitigation);  Ensuring Swaziland’s OVC QSS are adopted by relevant organizations;  Standardization of the PSS indicators to inform services to children and people living with HIV. 0.0% 25.0% 50.0% 75.0% 100.0% strongly disagree disagree neutral agree strongly agree The frequency of our support to beneficiaries has increased as a result of PACT’s direct support to our organisation (N=35) S Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 52 PACT’s FY2013 report describes how sub-partners contributed to the NSF (as shown in Table 16). In addition, all sub-partners working with OVC reported using Swaziland’s newly launched OVC Quality Standards. Table 16. Sampled sub-partner involvement in NSF 2014 to 2018 Thematic Area Sub-theme Sampled sub-partner Prevention  Social and behavior change communication (SBCC);  Condom use;  HIV prevention for most-at-risk populations (MARPs);  HIV counselling and testing (HCT)  VOC, SWABCHA  SWABCHA  SWABCHA  Cabrini, SWABCHA Treatment Care and Support  Pre-ART and treatment of opportunistic infections;  Antiretroviral therapy (ART) services;  Management of TB and HIV co-infection.  Cabrini  Cabrini, SWANNEPHA  Cabrini, SWABCHA Impact Mitigation  Food and nutrition security support;  Educational support for OVC;  Socialization and protection of OVC;  Psychosocial support;  Community systems strengthening for impact mitigation services.  Cabrini, Bantwana, LDS  SCSWD, LDS, Cabrini  LDS, SCSWD, Bantwana, Cabrini  LDS, Bantwana, SCSWD, Cabrini  Bantwana, SCSWD, Cabrini Figure 19 presents sub-partners’ responses as to whether PACT assisted their organizations to work effectively with government. Most were very positive about PACT’s assistance, and these views were also confirmed by key informant respondents. All sub-partners report that their work is aligned to government standards and guidelines, meaning that the sub-partners are following the standards and guidelines as set down by Government. This alignment is shown through several actions:  Some sub-partners are part of government structures working on standards and policy. For example, one sub-partner key informant said: “Rather than aligning, we’ve tried to help design where possible. We’ve contributed to the quality service standards…Rather than aligning we’ve tried to shape health care … where we can.”  Some sub-partners are part of national planning and TWGs. According to one sub-partner key informant: “The work that we do is it relevant…it is what national policies are trying to talk to…we are sometimes involved in the planning of national documents, like the national plan of action.”  The sub-partners providing OVC and prevention services have been trained by PACT on the standards and systems to which they need to refer in planning and implementing their work.  PACT worked with HIV counselling and testing sub-partners on adapting and implementing new guidelines and ensuring that that care and treatment sub-partners use the SNAP tools. One key informant noted: Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 53 “The SBCC strategy was aligned with NERCHA. PACT worked on it and used the strategy with all their sub-partners. It wasn’t perfect, but it (is) one example.” Government informants also acknowledge PACT’s work with government ministries and departments, and the sub-partners’ work with the government. In addition to aligning their work with government policies and standards, sub-partners also refer beneficiaries to government services, including clinics, hospitals, police, and social workers. Indeed, PACT actively encourages sub-partners to make referrals and to follow them up, and the referrals are included in the sub￾partners’ M&E data. Sub-partners also make referrals to other sub-partners and other NGOs. For example, various sub￾partners refer beneficiaries to SWAGAA counselling services. Another sub-grantee refers clients to SWAGAA, Social Welfare, the protection unit of the police, as well as to the Family Life Association of Swaziland (FLAS) which provides sexual and reproductive health services for patients. Finally, PACT’s assistance in linking sub-partners to government also reportedly yielded improvements in improved services as shown in Figure 20. Figure 19. The effects of PACT’s assistance to sub-grantees in working with government 0% 25% 50% 75% 100% Head of Organization Financial Manager M&E Officer Program Officer Other Grand Total PACT’s assistance helps us work more effectively with government (subpartners only N=14) Strongly agree Agree Neutral Disagree Strongly Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 54 Figure 20. Service effects of sub-grantee’s closer work with government CONCLUSIONS Around Sub-partner Geographic Coverage and Dosage:  There is no evidence to show that geographic coverage is related to a difference in the quality of services.  Increased coverage requires extra funding and can put a strain on organizations that are already struggling to diversify funding streams.  Attrition of high-quality staff is a concern for all the sub-partners and affects service delivery until suitable replacements are found and trained, regardless of geographic placement.  Developing standards for quality delivery helps set standards for service quality.  According to sub-partners, the emphasis put on dosage by PACT has improved their focus on increasing and strengthening the dosage and frequency of service provision, especially in prevention- and OVC-focused services. Around Engagement and Alignment with Government Systems:  PACT enables sub-partners to align their work with government by ensuring that sub￾partners understand what is required.  The OVC sub-partners align their services to national strategies and systems, and use the OVC Quality Standards which were initiated and developed by PACT and other organizations.  All sampled sub-partners were able to demonstrate that their work is aligned to the National Multi-sectoral Strategic Framework on HIV and AIDS.  A good networking and support system based on service provision has been built among sub-partners, which helps referral systems and processes, including referrals to government structures. 0% 25% 50% 75% 100% CEO, MD, or head of organisation Financial Manager M&E Officer Programme Officer Other Total PACT's partnerships with national structures help us to improve our services (N=35) strongly agree agree neutral disagree strongly disagree Source: Online Survey Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 55 RECOMMENDATIONS Around Sub-partner Geographic Coverage and Dosage:  PACT should assist sub-partners to find new ways of incentivizing staff so they stay longer in their jobs as well as succession planning, which affect the coverage and reach of services.  Developing common standards sets the way forward for quality service delivery; PACT and its sub-partners should actively support the development of standards in areas of service delivery beyond OVC, which is seen as important by the Swaziland Government. Around Engagement and Alignment with Government Systems:  PACT should encourage all sub-partners to prioritize participating in planning groups and TWGs to help develop and support high-quality national strategies and systems in the future which, in turn, will ensure their work is relevant and recognized.  PACT should encourage government to build on the referral system already in place between its sub-partners and develop a national referral network for service provision, which could link the four regions of Swaziland through service mapping. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 56 Question 5. What approaches for delivering sustainable community services for OVC are working well? Given the nature of this mid-term evaluation, the evaluation team determined that the best approach to determining what was “working well” was to obtain feedback directly from the beneficiaries and caregivers on the ground. As such their perceptions obtained through community-based FGDs is the basis for the findings and conclusions below. OVERVIEW Annual reports and workplans reveal that all 4 OVC organizations supported by PACT have a multi￾sectoral approach to supporting OVC, but each has a unique implementation approach and a unique package of services offered to their beneficiaries and constituents as depicted in Table 17. Table 17. Program Elements by sub-partners working with OVC (PACT funding only) Element of Response National Coverage Localized Coverage Save the Children Swaziland Bantwana Initiative Cabrini Ministries Lutheran Development Service Multi-sectoral approach (e.g. health, agriculture, social protection/development) YES YES YES YES Staffing  Community-based volunteers YES YES YES YES  Paid community-based care workers YES NO YES NO Point of Services Delivery  Communities / Households YES NO YES YES  Schools NO YES NO NO Education  Early Childhood Care and Development o Training of caregivers / production and dissemination of materials) YES NO YES YES  Education plans / ‘co-parenting’ NO NO YES NO  Grants for school fees YES NO YES NO  School supplies YES NO YES NO  Homework support / school enhancement NO NO YES NO Shelter  Shelter (boarding facilities / building houses) NO NO YES NO  Blankets / toiletries NO NO YES NO Food and Nutrition  Food provision o feeding scheme/ food parcels/nutrition supplements NO NO YES NO  Linkages with NCPs and WFP food programs YES unknown unknown YES  Monitoring results of food support (BMI monitoring) NO NO YES YES Agriculture and Economic Strengthening  Strengthening family agriculture o Agricultural inputs (vegetables /fruit trees /chickens/ bees /fencing) YES YES NO YES  Strengthening school agriculture NO YES YES NO  Strengthening community agriculture NO NO NO YES Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 57 Element of Response National Coverage Localized Coverage Save the Children Swaziland Bantwana Initiative Cabrini Ministries Lutheran Development Service  Income Support o Entrepreneurial skills training / credit and savings / vocational training) YES YES YES YES Health  Health services provision YES YES YES YES  Training of health care professionals NO NO YES NO  HTC / TB Screening NO NO YES NO  Treatment, care and adherence NO NO YES NO  Screening for all diseases NO NO YES NO  Referrals to other health services NO NO YES NO  Mental health services NO NO YES NO  MMC services NO NO YES NO  Community education and testing days NO NO YES YES  Health training for community based caregivers / volunteers YES YES YES YES  Home visits YES YES YES YES  Outreach / mobile health services NO YES YES NO  Transport for service provider to community / client to service point YES YES YES YES Child Protection  Training of child rights within community (traditional leaders/ guardians/ caregivers/ teachers/ OVC/youth) YES NO YES YES  Registration of legal documents YES YES YES YES  Property / land rights protection YES YES YES YES  Referrals (Police / MoH / DSW / civil society) YES YES YES YES  Family reunification NO NO YES NO Psychosocial support  Community-based needs assessment NO NO YES NO  Trauma / grief counselling YES YES YES YES  Life-skills training YES YES YES YES  Shelter (blankets/ toiletries) NO NO YES NO  Training of community caregivers / field officers YES YES YES YES  IEC materials YES YES NO NO  Peer support / youth clubs YES YES YES YES  Transport to services NO NO YES NO  Networking with partners / other service providers YES YES YES YES The following summarizes the results of the FGD scorecards and discussions using the 5 OVC service categories that are listed in the Standards for Quality Service Delivery to Orphans and Vulnerable Children in Swaziland (the Standards) as the basis for presentation of findings. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 58 FINDINGS Community ownership of any OVC program is central in the Standards and speaks to issues of sustainability. The introduction of programs must recognize all community stakeholders, including traditional authorities and representatives of local government structures, and the stakeholders need a clear understanding of what the program is trying to achieve to foster community ownership. The FGDs showed that communities are very positive about how the OVC programs were introduced (Figure 21). Discussions with beneficiaries reveal greater satisfaction with sub-partners who helped increase availability of, and access to, services through bringing services closer to communities. However, the FGDs also revealed that the respondents did not always know which organization was responsible for assistance; some services provided by other entities may have been attributed to the sub-partner because of similar mandates5 or because of effective linkages and networking. Figure 21. Response to the introduction of the projects to communities Satisfaction with OVC Services Delivery Average Satisfaction Scores for key OVC services are noted in Table 18. The services most appreciated by beneficiaries, with no statistically significant differences between them, are Agriculture support, Child protection and Psychosocial support. Table 18. Community-level Respondents Average Satisfaction Scores for OVC services OVC Services Ave Score for Service (of 5 possible points) Agricultural Support 4.10 Child Protection 4.08 5 For example, the Government of Swaziland is attempting to bring services closer to the community so that the acquisition of legal documents can be done at the Inkhundla level. And one PACT-supported OVC organization fetches WFP food from a depot. 78% 21% 1% 81% 16% 3% 93% 6% 1% 0% 25% 50% 75% 100% VERY POOR POOR AVERAGE GOOD EXCELLENT Manner of Program Introduction to the Community (i.e. was protocol observed) OVC Parent / Guardian Programme Volunteer / Care giver Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 59 Psychosocial Support 4.08 Health Services 3.76 Economic Strengthening 3.58 Education 3.29 Agriculture Support Agriculture support contributes to economic strengthening and food security. Pact issued a subgrant to TechnoServe as a specialized technical assistance provider for economic strengthening, including agriculture support, to support the 4 OVC partners. Agricultural support rated particularly high among programme volunteers. Among the 4 OVC programs, 3 supply agricultural inputs. These include vegetable seedlings, fruit trees, chickens and fencing. Seventy-three percent (73%) of respondents scored the provision of agricultural inputs as excellent or good. Only Bantwana supports school gardens, although Cabrini has a school garden which is utilized for agricultural education. Likewise, Bantwana has also provided some assistance around water storage for school gardens. School or homestead permaculture gardens scored highest among the services scored “excellent” by respondents at 86%. However, there were a few concerns about the sustainability of the entrepreneurial agriculture projects, especially the poultry projects, indicating that the community has not yet learned how to build up and sustain equity, especially around food. The agricultural activities were seen as making a substantial positive difference to food security and income generation, especially innovative approaches such as permaculture. According to the respondents, “The agricultural inputs were good quality. The way the children were taught to grow gardens is excellent and innovative. Parents are learning these methods from their children.” Child Protection and Legal Support Child Protection was highly rated particularly among OVC themselves and programme volunteers. The child protection elements that respondents see as most successful include (i) capacity building of NCP caregivers, who are trained and mentored by sub-partner staff and the Public Health Unit, (ii) access to HIV testing after sexual abuse is now considered by most community members as a right for the abused child, (iii) bringing police and other agents such as SWAGAA together to educate communities on the importance of seeking immediate help and healthcare, including post-exposure prophylaxis (PEP) for abuse survivors, (iv) access to PEP (where delivered), and (v) sufficiency of counselling for abuse survivors. The OVC Standards state that “the desired outcome for child protection and legal support is that Children are socially and legally protected from all forms of abuse and are meaningfully participating in issues affecting and involving them”. Acquiring a birth certificate for an OVC is essential as this leads to accessing grants and services from the government. In some areas 100% of respondents said the support for birth, marriage and death registrations was excellent or good, and that all sub￾partners involved in this activity are performing very well. The sub-partner with the highest score facilitates visits by government to the local Umphakatsi (an administrative subdivision smaller than an Inkhundla). Overall, respondents felt that support for survivors of child abuse was good at many levels, with two areas working well: grief counselling for OVC and sub-partner support for community-based caregivers, who are a crucial part of sub-partners’ work. The sustainability element of the work of Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 60 sub-partners involves ensuring caregivers receive ongoing training and support, plus some form of stipend. Certain sub-partners were worried that when more lucrative opportunities arise, caregivers are enticed away and therefore there are always vacancies to be filled. One sub-partner noted that recently incentives were being paid to community members to use their land to plant sugar, and this work already had depleted some of the caregiver workforce. Respondents were satisfied with the quality of training for caregivers, but not the frequency, and said caregivers need more educational materials for ECCD and medical supplies for sick people in the community. The approach of using community dialogues to advocate for the involvement of males in child protection was seen as excellent or good by 85% of respondents. The dialogues give communities better understanding of the issues and enable them to be more willing to take responsibility to uphold children’s rights. Psycho-Social Support (PSS) PSS is also highly rated OVC service, particularly among OVC themselves and programme volunteers. According to the OVC Standards, the desired outcome for psycho-social support is that “Children are integrated well within their families, friends and community and demonstrate self￾esteem and resilience”. Life-skills training for young people was scored at a high level across the board, with 88% of all respondents marking the intervention as excellent or good. The OVC pilot study also confirmed a substantial increase in life-skills training for children. Respondents were positive about the sub-partners’ training of parents, guardians, and traditional authorities to understand children’s psycho-social needs (particularly important in Swaziland which has many traditional norms and standards), and Community-based Child Protection Committees. However, many feel there are insufficient peer/youth support groups to provide additional PSS support. Health, Water and Sanitation Health services support received moderate satisfaction scores compared with the three services discussed above. According to the Standards, the desired outcome for health, water and sanitation is that the “Child receives health care services including: preventative care, curative care, promotive, and rehabilitation services”. Respondents were most satisfied with an outreach approach where the provision of transport is a key component as provided by Cabrini. Utilizing local clinics and partnering with other PEPFAR and Swazi partners were highlighted positively. The respondents’ satisfaction scores for health services shows that respondents were most satisfied with the support given by Cabrini through visits by community caregivers and access to medication, although access to medication was a weaker component of other sub-partners’ health services. Group education on TB/HIV for families or at homesteads delivered by Rural Health Motivators (RHMs), parish volunteers and other caregivers was thought to be excellent by 91% of the respondents. In the FGDs, the respondents noted an overall significant improvement in accessing health services, the support of caregivers, and the time to acquire medicine. But without adequate transportation there was a concern about the coverage of medical care, which was seen as unsatisfactory. The large rural areas of Swaziland generally do not have easy access to transport to clinics. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 61 Economic Strengthening Economic Strengthening beyond agriculture support received only moderate satisfaction scores, compared to the other OVC services. Sub-partners’ work with community members on economic strengthening was viewed as an essential part of building and strengthening the community as a whole and all 4 organizations provided training in economic strengthening or entrepreneurship. By and large, the training was viewed positively, especially the training provided by LDS, which partnered with TechnoServe Swaziland, and SCSWD, which used the WORTH model for savings and loans. . The Savings and Loans project was singled out as helping families and individuals save, even if they have very little. This service was provided by SCSWD, which has a large geographical footprint and can reach many communities through the staff based at their regional offices. As it is a successful project for strengthening the community, the opportunity for expansion is great, provided that funds and staff are available to run and monitor the training. Education Assistance Satisfaction with education support was rated lower than other OVC services. According to the OVC Standards, the desired outcome of Educational Assistance is that “the child is enrolled, progresses and receives Swaziland completion certificate at primary and secondary level or comparable level according to their ability”. As shown in Table 17, three sub-partners provide educational assistance: Cabrini provides a wide range of educational support for children in primary and high school. SCSWD, LDS and Cabrini support ECCD. Fifty-five percent (55%) of FGD respondents believed that provision of education grants was excellent or good, and community concerns around the grants have lessened now that primary school education in Swaziland is nominally free, although some schools do not comply as they insist that the government subvention is insufficient for them to run their operations. Also, most high schools add top-up fees to OVC grants. The education assistance approach used by Cabrini that emphasizes a relationship of ‘co-parenting’ between sub-partners and OVC and families was considered highly effective by respondents. Discussion groups expressed great satisfaction with ECCD activities as they believe these improve children’s access to primary school for which there are insufficient places. The OVC pilot study conducted by 9 organizations, including Bantwana, Cabrini and SCSWD, also reported an improvement in preschool enrolment based on the organizations’ use of the OVC standards (i.e. QSS). While Table 17 above shows that only a few sub-partners directly deliver educational support, all sub-partners facilitate education support from other sources. For many OVC and caregivers, the sub-partner is their main contact and advocate with potential educational sponsors. Shelter Cabrini is the only sub-partner providing shelter and material support. However this service is considered of the upmost importance by the FGD participants. Among the guardians interviewed, a number were grandparents, who said they could not cope with the number of OVC they had to shelter, clothe, and feed without Cabrini’s support. Cabrini offers boarding facilities to some OVC and builds houses for the most needy. By and large, the respondents were more satisfied with the quality of the shelter provided by Cabrini than with the quantity of the blankets or clothing provided. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 62 Food Support In the Standards, the desired outcome for Food and Nutrition is “Well-nourished children living in a food secure environment”. Caregivers at NCPs cook mealie meal and beans for OVC. In the case of Cabrini, food parcels are sent home with children during school holidays. Sustained food supplies have been achieved by sub￾partners working with neighborhood care points (NCPs), with the World Food Programme (WFP), and local stakeholders. Seventy two percent of the respondents scored this service as good or excellent. Also, 62% of the respondents felt that the provision of nutritional supplements was excellent. Sustainable Services Based on the key informant interviews, the sub-partners providing OVC services were not very optimistic about the sustainability of the services they provide without continuing funding support. Some felt that the community caregivers may continue to provide some support, especially those who are volunteers. The only service mentioned as sustainable was the cultivation of homestead gardens. The high mobility of teachers and students was noted as making the school services less sustainable. Most of the sub-partners said that the government would have to take over service provision without continued funding, but noted the lack of existing government structures to assist. CONCLUSIONS Agricultural support, Child protection, and Psycho-social support are the areas where beneficiaries and caregivers report the greatest satisfaction. Support for health services is also well received. However, educational support and economic strengthening can be improved at most organizations. The FGD discussions and the scorecard data point to the following approaches and OVC services that appear to work well:  Early Childhood and Care Development for enhanced OVC access to primary schools;  Overall access for OVCs to health care, medicine, and follow-up support, particularly where transport is provided;  The registration of births and acquiring of certificates to enable more OVCs to register and attend school more frequently;  Innovative agricultural technologies, such as permaculture, to make an impact on nutrition at schools and at the homestead level;  Savings and credit groups to help families save and to strengthen communities;  Life-skills training for young people to increase their awareness of needs and wants;  The awareness of child rights appears to be related to increased local condemnation of abuse and increased support (medical and psycho-social support ) of abuse survivors; In contrast, communities express the need for expanded supplies and services in the areas below. It should be noted that many of these are not a focus area of Pact sub-partners due to concerns around sustainability of providing material supplies and incentives, and that others are beyond the control of community organizations, such as the inconsistent food supply at Neighbourhood Care Points.  Expanded educational supplies to OVC; Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 63  More consistency of the food supply at feeding points;  Improved timing of the response by police to abuse survivors;  Enhanced confidentiality in health facilities to ensure confidentiality on the HIV status of OVCs and others;  Expanded entrepreneurial training;  Expanded training on savings and credit schemes;  Increased supplies for caregivers such as soap, Vaseline, medication, and food supplements.  Incentives for caregivers, such as uniforms and umbrellas. RECOMMENDATIONS  Continue the current model of involving community stakeholders, building trust and tackling basic needs.  Continue the approaches listed as working well in the Conclusions, above, and intensify them were possible.  Increase linkages with other organizations so that more services are brought to the community, whether this is a health service such as bringing mobile clinics to schools, or facilitating government officers to register certificates. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 64 Question 6. To what extent has PACT strengthened efforts to improve the OVC response beyond sub-partner service delivery through activities related to developing of standards and other national processes? FINDINGS During the REACH project, PACT has developed strong working relationships with government departments, including the Department of Social Welfare (DSW), NERCHA, SNAP, NCCU, and the MoET. With each department, there is a different relationship depending on the need and focus. PACT acts either as an initiator of an idea and process which is then accepted and followed up by the government department, or participates as a member of a TWG where it is a contributor to the government discussion and debate. In either capacity, PACT needed to build a good level of trust with government before its role was accepted, which it has done throughout its work in Swaziland. The various national processes and standards in which PACT participated are described below. Swaziland OVC Standards PACT was an initiator of the Swaziland OVC standards – known as the Standards for Quality Service Delivery to Orphans and Vulnerable Children in Swaziland – which helped OVC service providers to strengthen and measure the quality of their work. Working through the offices of the DSW and NCCU, PACT collaborated with a range of stakeholders to develop, pilot, and launch the standards, which were officially launched in November 2012. Acknowledgements in the front of the standards document by the Principal Secretary in the Deputy Prime Minister’s Office state: “Special thanks also go to PACT for their technical input throughout the process of developing the quality service standards.” Key informants at government departments talked positively about PACT’s assistance in this area and see PACT as a partner in their ongoing work to finalize policies and standards. A formal pilot of the OVC standards was completed between January and November 2011 under the auspices of the NCCU with technical assistance from PACT and supported by PEPFAR. Three of the 4 OVC sub-partners were involved in piloting the standards and giving input on their usefulness and applicability. The pilot informed the standards by: 1. Ensuring that standards were understandable and feasible. 2. Documenting what steps organizations needed to take to implement and adhere to the standards, and what best practices emerged. 3. Determining if implementation of standards improved quality of programming and services delivered and lead to a measurable difference for children. The data collected through the pilot study found substantial increases in some OVC service results. For example, 3,012 children were followed through the study (from a baseline of 3,554), and the quality of some services improved significantly through the use of the standards, with many more children benefitting from these services (see Table 19 for examples). The pilot showed that the quality of service delivery could be affected by the use of the OVC standards. Through the support of government, the standards are now well established and used by the OVC service providers across Swaziland. One key informant said, Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 65 “[PACT] came to [us]...we talked about a nutrition program…where we can improve as an organization so that we can be in line with the nutrition standards. The [standards] help us to better deliver the nutritional service in the schools.” Table 19. OVC Standards pilot study key results National Strategic Framework for HIV/AIDS PACT has helped, and is still helping, government officials to review the National Strategic Framework (NSF) for HIV/AIDS, and its work to support the expansion of the NSF has been well￾recognized. One government key informant said, “…the NSF was a clear example of PACT’s strong voice on OVC and in the prevention area.” Swaziland Civil Society Priorities Charter PACT also participated in the development of the Swaziland Civil Society Priorities Charter, which was launched in November 2013. The Charter names OVCs as a key population for services, especially in the Lubombo region, and it lists ‘protection and creating a conducive environment’ and ‘access to services’ as priority areas for all key populations. Save the Children Swaziland (SCSWD), Bantwana Initiative, and Cabrini Ministries are named as OVC service providers in the Charter, which is another example of how PACT has helped to strengthen the response to Swaziland’s OVC crisis. Swaziland Early Learning and Development Standards (SELDS) During 2013, PACT also was part of a technical and advisory group that assisted in producing the SELDS which were launched in March 2013. These standards assist caregivers, parents/guardians and others to monitor the development of children, aged 0 to eight years. The standards are essential as a guide for the well-being of Swaziland’s growing group of vulnerable children and are being used by the MoET. PACT has been asked to participate in a multi-stakeholder forum set up by the MoET, which brings together all the government departments so the needs of children are addressed in a holistic manner. PACT’s close association with government departments is also acknowledged by other agencies working in Swaziland. A key informant at another agency said that PACT was now seen as an essential partner for government at many levels and stated: “[PACT has]… been working closely with the National Children’s Coordination Unit, helping the psycho-social support program…including the national strategy that was developed which has been rolled out.” mHealth Initiative A recent initiative by PACT with government has been to introduce mobile data collection technology (mHealth) .as a tool for sub-partners to use to collect survey and routine data. Although still in a nascent form, PACT is trying to get mHealth and associated software introduced across and used by all sub-partners. Pilot projects using mHealth technology among 3 of the sampled sub￾partners were introduced in FY2013 and FY2014 as follows: Service category Baseline Endline Children who had never attended school 34.4% 18.3% Children enrolled in pre-school 32.4% 44.3% Children who received birth certificates and other legal documents 40.8% 51.9% Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 66  SCSWD conducted a Child Profiling Survey which identified and mapped the needs of 10,500 children;  LDS profiled 67 NCPs; and  SWABCHA planning to collect end-of-project evaluation data and routine HTC data. PACT is keeping government abreast of how the technology is affecting service delivery and helps the sub-partners collect, interpret, and use data effectively. To summarize, PACT’s work with government includes:  Technical advisor and collaborator on standards for OVCs;  Supporter of NCCU on SELDS;  Collaborator on Swaziland Civil Society Priorities Charter;  Member of various technical working groups (TWGs), including those on OVC, PSS, Child Protection and Early Childhood Care and Development (ECCD);  Advisor to NCCU and member of committee standardizing quality indicators for psycho￾social support of children; and  Through the NCCU, part of the Child Protection Network. Through its work with government, PACT provides capacity building at the system level, influencing and advising on policy and processes. A spin-off of PACT’s capacity building at the system level is that some sub-partners have grown capacitated and confident enough to interact with government directly and assist government staff in their work with children at the local level. For example, one sub-partner elected to help under-resourced, scarce in number, social workers with their transport needs. This led to a more collaborative response by the DSW to the needs and subsequent help for children. Respondents from the online survey confirmed that PACT’s work with government departments helped to improve their service delivery (Figure 20). This confirmation is a positive reflection by the sub-partners on PACT’s work with national structures and is in line with other comments received. Finally, a government key informant reported that PACT has had a major influence on government services and has raised the bar in terms of getting various departments to respond: “Concerning PACT…the interest they’ve [shown] has been high…for the OVC response, which was non-existent for a while…at a national level.” CONCLUSIONS  PACT is well-positioned with government departments and is respected for its technical know-how on matters relating to OVCs through its support of the initiation, piloting and development of the OVC quality standards that are positively affecting the OVC service delivery.  PACT is invited to sit on various TWGs to offer insights and support for government’s work.  The work that PACT has completed with government has had a positive influence on sub￾partners’ work and has positioned them well to continue to work with government when the REACH Project ends. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 67 RECOMMENDATIONS  PACT should continue to work until the end of the REACH project with government departments, both initiating and responding when appropriate, in the development of standards and other national processes relating to children, especially OVCs, and HIV/AIDS. This work will be a legacy of PACT’s and USAID’s support to the Government of Swaziland.  PACT should continue piloting work with mHealth technology, which is a programmatic innovation that also could reduce costs and labor hours for service providers and government officials alike. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 68 Question 7. What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? FINDINGS PACT’s 2013 Annual Progress Report (APR) reflects on the work it has undertaken with CANGO. As noted in the discussion to evaluation question 1, PACT’s support to CANGO has focused at two levels: first, to build the internal capacity of CANGO as an organization operating as a networking and advocacy entity and second, to build CANGO’s capacity as a UGM. These two objectives are different yet complementary, and require substantial support from PACT to ensure that CANGO will be able to meet both objectives effectively. To assess how CANGO is progressing, PACT uses a tool called the OD Roadmap to measure progress and areas of concern. The evaluation team administered the OD Roadmap by interviewing CANGO senior staff who reflected on their work and the organization as a whole, and to identify continuing issues and concerns, plus areas of growth. CANGO staff were asked to rate themselves according to a scale of 1 to 5 for several subsets of the tool. Table 20. CANGO OD Roadmap – Strengths and Weaknesses as rated by CANGO OD Roadmap categories Strengths Weaknesses (Gaps) Purpose and Planning CANGO has a documented strategic plan reflecting stakeholder inputs that is presently being revised; a work plan is in place with specified objectives; SMART Outcomes Objectives linked to indicators has been finalized. Recruitment strategies are in line with the law; all staff job descriptions have been finalized. Work plan needs to be monitored more regularly; staff need to be updated on vision and mission; a staff talent management plan is needed; need to rethink resources to keep high level, quality staff. Human Resources Management Code of ethics in policy document; salary deductions are in line with policy and the law; annual staff assessments are conducted; comprehensive staff files are in place. HRM policy document needs further work, especially on disability policy; comparative salary scales with similar organizations are not in place. Monitoring, Evaluation, Reporting and Learning Good allocation of budget to MERL; all tools for M&E in place; DQAs held regularly; reports sent to PACT on time. Data management plan is still under development and needs to be finalized; data backed up weekly by a consultant. Networking MoU framework in place with DPM’s office on food security; chairs program section of the NERCHA Council; participated in NSF response management cluster. More MoUs needed with government departments; not strong in sharing lessons learned on their UGM experience. Governance Registered as a NGO with all necessary statutory considerations; board in place and quarterly meetings held – decisions shared with staff; good supervision of staff in place; participative leadership is evident; good horizontal and vertical communication systems in place. No succession plan in place; a review of policies that guide board functions is needed; organizational chart to be finalized; staff reward system to be reviewed. Organizational Sustainability Good resource mobilization strategy in place; staff have RFA skills; diversified Updating of resources mobilization strategy is needed; response to call for proposals needs Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 69 OD Roadmap categories Strengths Weaknesses (Gaps) income in place; proactive in setting up meetings with government and donor agencies; good marketing strategy and outreach using multi-media; easily recognizable logo. to be pursued; one major funder (PEFPAR); funds for core operation costs have not been set aside for the next 12 months; plan needed to guide donor interaction. Financial Management Detailed line item budget in place; monthly management reports completed on time; annual audits in place; assets register; two out of three signatories for payments and cheques. Explanations on budget variances to be improved; follow-up with auditors for speedier service is needed; need to update Pastel system as it does not allow accruals; present system cannot generate reports in multiple currencies; VAT systems need to be reviewed and improved further. Grants and Compliance Is very compliant with regard to awards and understanding of the grant at an internal level helping them to manage others; most staff are aware of grant allowances’ controls and compliance issues; burn rates are within 20% of projections and reports are submitted on time and are accurate. Not all staff understand the grants controls around allowable and unallowable expenditures; timesheets are not well recorded – major work is needed in this area; filing systems are in place but need to be updated. Operations Management Has statutory insurance in place for staff; insurance for assets in place; security company in place for 12 hours a day and has an alarm system; not able to always operationalize procurement competitive bidding process. IT infrastructure not fully stable due to electricity outages; lack of internal staff person to manage data back-up. UGM & Capacity Development Sub-partners selected through transparent process that is confidential; MCAT completed with 3 new sub - grantees; CANGO has helped sub￾grantees complete work plans and project designs; different approaches used for capacity development of each sub-partner. Sub-partner handbook to be developed; not yet transitioned from co-facilitation with PACT; increase of sub-partners for CANGO puts pressure on time allocated to effectively support each one; peer reviews needed but to be completed by CANGO on its own; work on budgets and burn rates to be completed for sub-partners; grant close-outs to be looked at urgently. Of the 10 Roadmap areas, 9 are concerned with CANGO as an organization and 1 with its role as a UGM. As the Roadmap illustrates, and with reference to CANGO’s ISP of October 2012, capacity building of CANGO internally is as important as building its role as a UGM and some major successes have been made. One key informant noted: “There were no M&E systems for CANGO, before this project....but with assistance from PACT…now put in place M&E systems for the grants management unit and for CANGO as an organization.” PACT’s APR FY2013 shows that the overall score for the Roadmap assessment has moved from an average score of 1.85 across the 10 sections in 2012 to 2.89 in 2013. The capacity of CANGO to operate as a UGM has increased significantly to 2.46, to which PACT’s capacity building, training, support, mentoring, peer reflection, and re-training, etc. has contributed. The evaluation team assessed the Roadmap and the ease in administering it with CANGO staff. It is a comprehensive tool that identifies areas of performance and gaps, but there are some concerns: Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 70  The OD Roadmap is very dense and is more useful with senior staff. More junior staff in an organization may struggle to understand the nuanced differences between the levels (1 to 5). This means that team scoring is not possible without a lot of support from the senior staff.  The section on UGM and Capacity Building does not separate concepts at the highest level, and does not address the processes for achieving these levels. In contrast, the ISP which is developed from the OD Roadmap is more comprehensive, gives detailed and clearly stated action plans with a timeframe, and sets down the priorities and who is responsible for which action. However, CANGO needs better internal systems to track what its sub-partners are doing as it works through the actions set down in the ISPs. The ISP further reveals that CANGO needs to work on building its capacity to be a UGM, but also must not neglect its role as an organization that networks and advocates. In its role as a central networking organization, CANGO needs to keep in contact with the NGOs in its database and better coordinate their networking with each other. Perceptions about CANGO’s effectiveness as an UGM vary according to the maturity of the sub￾partner under CANGO’s oversight. The more established sub-partners believe that CANGO is slow in responding to their requests. According to several key informants, on many occasions CANGO first checks with PACT before it takes action. It is understandable that while CANGO is in a growth situation, it is still refers some sub-partner issues to PACT, but it means that CANGO’s expertise as an UGM is far from accepted by all the sub-partners. But many sub-partners do see some positive changes in CANGO’s performance as an UGM, citing that CANGO is now better skilled and better able to cope with the stresses of managing sub-partners. Yet there is still a belief by sub-partners that CANGO needs further capacity building before it can become a fully competent UGM. In re-administering the OD Roadmap, it was clear that senior staff at CANGO are focused on their areas of expertise and do not necessarily share their strengths and weaknesses with their colleagues. This indicates that more teamwork is needed to improve understanding of the overall manner in which CANGO is being managed and developed. In its weekly team meetings, CANGO should ensure two standing items on the agenda: one on internal capacity building and the second on UGM capacity building. There should be a clear demarcation between the two. CANGO itself is very positive about the OD training and support it has received from PACT, emphasizing:  Support by PACT to their leadership has strengthened the organization  Assistance by PACT to sub-partners to fulfil their grant mandates has been consistent and has helped CANGO  Mentorship and the coaching by PACT has strengthened CANGO’s systems  Strategic planning support has included mapping objectives, the results needed and program indicators, which has helped CANGO to focus more efficiently During 2013, CANGO had a staff complement of 9, but this appears to be insufficient to manage all the work. There is a definite need to increase its human resources capacity, especially with 3 new sub-partners to manage which has brought extra processes into play for CANGO. CANGO itself expressed some reservations as to how it will support all sub-partners with its present staff complement and skills in order to show results: “We’re trying to focus on these three new partners...I think we’ve made some progress, but it is obviously not as fast, the progress, because when implementing the ISP, the Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 71 initial one has many action items. You start with the easy ones and you do them, but the more difficult ones and the ones that take time, the ones that are slower in implementing...the progress I feel is slowing down because the easier ones and the ones that take less time have already been implemented.” CONCLUSIONS  The OD Roadmap score shows that CANGO is ‘emerging’ as an UGM based on a 65% improvement over its baseline, but more work is needed to build the capacity of CANGO as a fully operational UGM including continuing support of its leadership.  Pact’s support of CANGO to become a UGM is producing results, but the organization cannot yet operate as an UGM on its own.  Some sub-grantees are not yet confident in CANGO’s organizational abilities.  Human capacity within CANGO needs to be expanded and stabilized.  CANGO understands the OD Roadmap, but needs more guidance on how to score themselves. Although the Roadmap is comprehensive and allows CANGO to see how it is developing, it may not be the right tool for measuring CANGO’s capacity development.  CANGO still needs help in guiding sub-partners through their ISPs and action plans.  It seems unrealistic to expect CANGO to be able to manage the technical capacity building of PEPFAR sub-grantees without additional support, as the sub-grantees provide a wide range of HIV/AIDS services outside the purview and expertise of CANGO’s small UGM staff. Recommendations  A clear division between activities relating to internal capacity building of CANGO as an organization versus activities relating to capacity building of CANGO as a UGM is needed.  Senior staff within CANGO need more mentoring to allow leadership to emerge quickly, for succession planning, and to bring CANGO to independence as an organization. The team also needs more bonding between each other and more knowledge about all their areas of work, so they are able to support each other and discuss their work at many levels with sub-grantees and government officials.  CANGO leadership needs to recommit to ensuring that government entities, sub-grantees and internal CANGO staff are aware of the strategy and timelines going forward to becoming a functioning UGM as quickly as possible.  CANGO needs help in identifying suitable staff to further grow its organization. It also needs to increase its budget to employ and retain these staff.  CANGO staff need refresher training on their use of the OD Roadmap and to understand the difference between the various levels and what these mean practically to their organizational growth, or another tool should be adopted. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 72 Question 8. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? FINDINGS PACT’s REACH Project has two objectives focused on building the capacity of civil society organizations in Swaziland to manage Global Fund resources:  Objective 1: To strengthen 1-2 local NGOs in umbrella grants management thereby enabling them to manage and sub-award funding from PEPFAR, Global Fund and other major donors in Swaziland.  Objective 4: To train, mentor and provide and/or link Global Fund civil society recipients to appropriate technical assistance. To meet objective 1, PACT has built the capacity of CANGO in sub-recipient management, which is a Global Fund capacity requirement. The focus on building CANGO’s capacity to manage sub￾recipients was based on CANGO’s selection by the NERCHA, the Global Fund principal recipient (PR) in Swaziland, to be a super sub-recipient (SSR) in Round 7, Phase II Global Fund resources managed by NERCHA. CANGO signed an agreement with NERCHA to manage other civil society sub-recipients of Global Fund resources in Swaziland. However, NERCHA’s Grants Management Unit (GMU) was unable to provide ‘hands-on’ support to CANGO in managing sub-recipients; hence the importance of PACT’s training of CANGO in sub-recipient management. Moreover, CANGO now manages 8 PACT sub-partners who receive PEPFAR funds, underscoring the importance of strengthening their capacity in sub-recipient management. PACT has helped CANGO to widen its resource base which should help the organization with its future sustainability. However, CANGO’s umbrella grants mechanism staff is small, without a range of expertise in HIV/AIDS technical programmatic areas. PACT also trained CANGO in financial management and monitoring and evaluation, two other Global Fund capacity requirements, in workshops with its other sub-partners. Primarily, though, PACT has used its own capacity development tool, the ‘OD Roadmap,’ to build CANGO’s organizational capacity in umbrella grants management in 10 areas: purpose and planning; human resources management; monitoring, evaluation, reporting and learning; networking; governance; organizational sustainability; financial management; grants and compliance; operations management; umbrella grants management and capacity development. The tool focuses on strengthening CANGO as an organization as well as strengthening its umbrella grants management capacity. Yet it does not include risk management or much focus on procurement and supply management, two other important capacity requirements of the Global Fund. Thus PACT’s OD Roadmap organizational capacity-building tool is too limited to meet all the capacity requirements for Global Fund sub-recipients. The Global Fund sub-recipients managed by CANGO to date include SCSWD, SWABCHA, the Alliance of Mayors’ Initiative for Community Action on AIDS (AMICALL), and Church Forum. SCSWD and SWABCHA also receive PEPFAR funding through PACT, although SWABCHA’s funding from PACT ended in September 2013. Unfortunately, the Global Fund’s resources available to these organizations and many other organizations in Swaziland ended in December 2013, except those providing treatment services, such as the Swaziland National Network of People living with Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 73 HIV/AIDS (SWANNEPHA), another PACT sub-partner whose funding from PACT ended in September 2013. Wider Global Fund support to Swaziland is expected to resume in 2015 through the Global Fund’s New Funding Model (NFM). In its work toward meeting all the REACH project objectives, PACT has conducted capacity-building workshops for all its sub-partners on Global Fund capacity requirements around financial management and M&E. These workshops have focused primarily on managing PEPFAR-funded programs and monies in Swaziland, yet they also have built the capacity of the participating organizations to manage other donor funding, including from the Global Fund. Table 21 below presents the list of the PACT workshops relevant to Global Fund requirements. Table 21. PACT sub-partner capacity-building workshops FY2011-2013 relevant to Global Fund capacity requirements Workshop Date Workshop Focus No. of NGOs Participating December 2011 Financial Management 14 February 2012 Basic Monitoring and Evaluation 13 May/June 2012 Data Quality Management 14 August 2012 Basic Excel and Data Analysis 14 November 2012 Data Quality Management Training 7 December 2013 Basic MER Training 8 December 2013 Finance Training 8 PACT’s capacity building in these 2 Global Fund areas has been an advantage to all organizations that have received this training, all of which may be future sub-recipients of Global Fund resources in Swaziland. Thus, PACT’s capacity building approach in specific areas relevant to Global Fund requirements can be seen as a strength. On the other hand, PACT has not provided capacity building for its non-CANGO sub-partners in other Global Fund requirements, i.e. in procurement and supply management, sub-recipient management, or program management (although a program management workshop was planned by PACT for February 2014), or risk management, a growing focus for Global Fund sub-recipients to understand and build into their programming. Thus PACT has not taken a comprehensive approach in building the capacity of its sub-partners to become potential Global Fund sub-recipients in the future, which can be seen as a weakness. Nonetheless, PACT has provided some training for potential future Global Fund sub-recipients that has not been provided by NERCHA, such as M&E and more extensive financial management training. PACT engages with NERCHA staff members on a regular basis, but mostly with its impact mitigation coordinator related to PACT’s support for its sub-partners’ OVC programming. PACT also has worked with NERCHA on reviewing, expanding and extending Swaziland’s NSF 2014-2018, as have PACT’s sub-partners. PACT staff and its OVC sub-partners assisted the writing of the section of Swaziland’s Global Fund proposal on the need for OVC programming in Swaziland, a contentious issue at the time given the Global Fund’s cost-benefit analysis for providing resources for OVC support in the country. PACT also interceded successfully with NERCHA on behalf of SWANNEPHA when the organization’s funding from the Global Fund was decreased. NERCHA considers PACT to be an important partner on a number of initiatives, including its capacity building of CANGO to be both an umbrella grants mechanism and a Global Fund super sub￾recipient. According to a key informant at NERCHA, Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 74 “PACT has done a really amazing job in capacitating CANGO. What our interests would be in going forward is to also bring on board other partners…The other partners also rally behind CANGO and support this capacity building. It’s now operating like any other organization. So, it’s been a wonderful experience for me with CANGO. Going forward let’s get more. Let’s get more partners to do the same.” As part of its capacity building and strengthening of CANGO to be a super sub-recipient of the Global Fund, PACT engaged NERCHA in meetings with CANGO to discuss issues arising around resource disbursement, which was a problem noted by many key informants. Thus PACT liaised with the two organizations and arranged joint meetings to alleviate some concerns and to strengthen the collaboration to ensure harmonized support for CANGO’s UGM. PACT and most of its sub-partners also participated with NERCHA in developing the “Swaziland Civil Society Priorities Charter: An Advocacy Roadmap for the Global Fund to Fight AIDS, Tuberculosis and Malaria New Funding Model”, initiated by AIDS Accountability International, an NGO based in South Africa, with support from the Ford Foundation. The Charter identifies six priorities for Global Fund support through the NFM in the future: behavior change; treatment, care and support; condom promotion; key populations; PMTCT; and MMC. CONCLUSIONS  PACT has conducted 7 workshops for its sub-partners relevant to Global Fund capacity requirements for sub-recipients. However, these workshops have focused on only 2 of the Global Fund’s 4 capacity requirements for sub-recipients. Thus there is a need for additional capacity building of potential Global Fund civil society sub-recipients in Swaziland in the Global Fund’s capacity requirement areas.  CANGO needs additional strengthening of its Global Fund capacity requirements as a super sub-recipient, especially in the areas of procurement and supply management and risk management, as introducing risk reduction into programming and the grants process has become an increasing focus for Global Fund sub-recipients.  PACT’s liaising with NERCHA on Global Fund issues has had some positive effects on civil society organizations, but there is an even greater need for liaising with NERCHA and advocating on behalf of civil society organizations for future funding since most of the Global Fund’s funding in Swaziland ended in December 2013. RECOMMENDATIONS  PACT should prioritize ensuring CANGO’s involvement in the negotiations leading up to and the drafting of the concept note to the Global Fund, emphasizing the need for funding to civil society sub-recipients through the Global Fund’s NFM in 2015.  REACH Project funding is due to end in March 2015. If resources are available, PACT could focus on additional capacity building of civil society organizations to become potential Global Fund sub-recipients. Bringing Global Fund capacity-building experts to Swaziland may be the most effective approach to take. No matter how it best can be done, the opportunity to further strengthen civil society organizations toward becoming potential Global Fund sub￾recipients should not be missed.  Likewise, CANGO would benefit from additional capacity building over the coming year in Global Fund sub-recipient and PEPFAR sub-partner management, especially in the areas of procurement and supply management, an essential Global Fund capacity requirement, and in Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 75 risk management, which is receiving additional emphasis by the Global Fund in 2014 to help sub-recipients understand how to reduce risk in programming. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 76 OVERALL CONCLUSIONS Based on nearly all the data and information collected and reviewed by the team, including all the KIIs and FGDs, PACT has done a commendable job in building the organizational capacity of its sub￾partners in Swaziland, although there are some gaps identified by the team that normally are included in organizational development. These gaps include a lack of focus on leadership training for senior management within organizations, and team building involving the whole staff at organizations early in organizational capacity development training. These areas are integral to organizational cohesion, mutual support, organizational esteem, and the ability to embrace change positively. The ability of organizations to grow and truly thrive emanating from the capacity building conducted by PACT varied based on a wide variety of factors, including how long the organization has been in existence, how developed it was prior to PACT’s capacity building, as well as the internal impetus for organizational change and evolution. During the period of PACT’s capacity building, sub-partners experienced organizational changes that had positive and negative effects overall, including staffing changes, funding level increases or decreases, and changes in programmatic focus. Such changes affect organizational progress. Likewise, these changes also affect an organization’s capacity and delivery of technical HIV/AIDS services, including OVC services. Organizations change over time based on both internal and external factors; these factors affect both the potential for positive change and the pace of change. Thus building the capacity both organizationally and technically in HIV/AIDS service delivery is both a difficult endeavor and a complex process as it has to be done amidst other fluctuations affecting organizational stability. According to the KIIs, FGDs and results reviews conducted by the team and the anonymous online survey, PACT has accomplished more in its organizational capacity building than in its HIV/AIDS technical capacity building during the period FY2011 through FY2013. This conclusion is not surprising as PACT placed greater emphasis in its range of capacity building on organizational capacity building than on technical HIV/AIDS capacity building, reflecting the overarching goal of its cooperative agreement “to build strong, well-functioning civil society organizations in Swaziland.” Moreover, PACT sub-partners were expected to receive technical support on HIV/AIDS services from other PEPFAR partners considered to be the technical experts in their respective technical areas. In summary, each organization interviewed by the evaluation team underscored the importance of PACT’s organizational capacity building to their present stability, and all were interested in continuing support from PACT in their organizational development. Most also said they would like additional technical HIV/AIDS capacity building in the future, at least some refresher training. As the HIV epidemic continues to have a heavy impact in Swaziland, further strengthening of the overall response to the epidemic by civil society organizations is needed. OVERALL RECOMMENDATIONS PACT should refine its organizational capacity-building model and centre it on organizational development as the foundational overarching theme under which the full range of capacity-building activities occur. This will require a more intensive focus on staff skilled in organizational development along with adequate budgetary resources. The sustainability of organizations is based on strong and continuing leadership; without this, the effect of any capacity building initiative will be lessened. Thus in future capacity-building awards, the selection of organizations for capacity building should begin with an assessment of senior Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 77 management leadership. Likewise, no matter how strong senior management leadership appears, it requires a special focus for capacity building to ensure continuing progress and sustainability of the organization. Programme managers also would benefit from leadership and supervision training and coaching. There is a continuing need in Swaziland for continued organizational capacity building of CSOs, but there is also an urgent need for technical capacity building. Expanding the technical HIV/AIDS skill set and beneficiary reach of CSOs in Swaziland should be the priority for USAID and PEPFAR resources over the next five years, in order to scale-up Swaziland’s response to the HIV/AIDS epidemic to achieve an “AIDS-free generation”. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 78 ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK PROJECT TO BE EVALUATED Project name: Rapid and Effective Action Combating HIV & AIDS (Community Reach) in Swaziland Cooperative Agreement No.: 674-A-00-10-00050-00 Project Dates: March 2010 – March 2015 Agreement Value: $17.5 million Implementing Organization: PACT 1. Objective of the evaluation The objective of this mid-term performance evaluation of PACT’s Community Reach Project in Swaziland is primarily to assess the quality of project implementation, to determine which approaches and activities are working well and why, and to indicate areas that may require mid-term corrections. Additionally, findings and recommendations from this evaluation may also inform future USAID solicitations in the area of local institutional and technical capacity building, as well as the area of Orphans and Vulnerable Children (OVC). Shortly after the midpoint of implementation of the Community Reach Project, there is a need to assess progress that has been made by the project toward achieving its set goal, objectives, expected outputs and/or outcomes. In terms of funding volume, PACT is PEPFAR/Swaziland’s largest partner in its Orphans and Vulnerable Children (OVC) portfolio. PEPFAR countries are urged to invest in building evidence to assess effectiveness and impact of their OVC programs. Thus, this evaluation will also include a focus on PACT’s effectiveness in strengthening technical and institutional capacity for quality OVC service delivery. The audience of the evaluation report will be PACT as implementing partner, the Coordinating Assembly of NGOs (CANGO) as the local Umbrella Grants Management Partner, and USAID/Swaziland and PEPFAR/Swaziland, as well as USAID Southern Africa and the Global Health Technical office in USAID/Washington. 2. Background of project Community REACH in Swaziland is a five-year project managed by PACT and funded by USAID/Swaziland with project duration from March 2010 to March 2015. The project is a successor project to Community REACH 1 which from 2005 to 2010 provided grants to organizations providing HIV services in Swaziland. The current award continued with activities under the previous agreement and added new activities with capacity building of a local organization (CANGO) to become an Umbrella Grants Manager for civil society organizations. The program was designed to facilitate the efficient flow of grant funds and to deliver targeted technical assistance and capacity building services to organizations contributing to the HIV and AIDS response. The overall goal of this award is to reduce the impact of HIV & AIDS and improve health care for the people of Swaziland by developing a strengthened, coordinated civil society response to HIV & AIDS. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 79 In order to achieve this goal, the project is expected to achieve the following results: Result 1: To strengthen 1-2 local NGOs in umbrella grant management thereby enabling them to manage and sub award funding from PEPFAR, Global Fund and other major donors in Swaziland. Result 2: To provide PEPFAR funding and minimal technical assistance to 3-5 mature NGOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 3: To provide PEPFAR funding and a full package of technical assistance to 10 – 15 nascent NGOs/CBOs/FBOs to enable them to deliver reliable and quality HIV & AIDS services to the community. Result 4: Provide an as yet to be determined number of Global Fund recipients with training, mentorship and linkages to appropriate technical assistance. USAID/Swaziland supports programs focusing primarily on prevention (sexual prevention, PMTCT, male circumcision), impact mitigation with a focus on vulnerable children and human and institutional capacity development (capacity building for community-based programming, strengthening of health systems, institutional strengthening of local NGOs). The HIV response in Swaziland is seriously hampered in all program areas by limited human and institutional capacity. The PACT award was designed to address capacity gaps in civil society organizations providing community services. PACT operates as an umbrella grant mechanism with a focus on strengthening the organizational effectiveness of NGOs to improve their capacity in project management, financial accountability, and monitoring and evaluation for results-oriented programming and organizational sustainability. PACT’s mandate is to build the capacity of organizations in a number of critical areas including grants and finance management, technical and program management, organizational development, monitoring, evaluation, reporting and learning. PACT uses capacity assessments to conduct in-depth reviews of each partner’s strengths and weaknesses in defined areas, based on which an institutional strengthening plan (ISP) is developed which details planned interventions to boost capacity in specific areas. PACT staff provides on-site assistance to address organizational weaknesses as well as customized group trainings to address institutional capacity needs identified across partner organizations. PACT currently supports 14 partners delivering HIV services in various technical areas which include sexual prevention, HIV testing and counseling, care and treatment, TB/HIV, impact mitigation, and gender. Some of PACT’s sub-partners operate in all four of Swaziland’s administrative regions, while others have a more limited reach (see table below and attachment “List of Communities supported by PACT Partners”). In FY2012, PACT partners reached over 58,000 beneficiaries (Sexual prevention 13,083; Counseling/Testing 3,398; Care including OVC 58,094; Clinical care 19,930; ART treatment 2,353). PACT also supports the Coordinating Assembly of NGOs (CANGO), in developing its capacity to serve as a local Umbrella Grants Mechanism (UGM) for civil society organizations in Swaziland for other donors, with a specific focus on the Global Fund under the National Emergency Response Coordinating Agency (NERCHA). NERCHA lists CANGO as a ‘super-sub-recipient’ tasked to serve as an umbrella body for civil society organizations. Five PACT sub-partners have shifted over the last 15 months to be managed directly by CANGO (see table below) with close mentoring and supervision from PACT. In FY2013 PACT is supporting CANGO to undertake a competitive selection process for two to three new sub-partners. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 80 A summary of partners’ technical areas and geographical focus is shown in the table below. The partner table also shows the current rating of each NGO as per PACT’s capacity development stage rating, defined as follows: Nascent All capacity areas measured are in rudimentary form and organization does not yet have systems or processes in place. The organization may be flexible and full of energy. Emerging Some capacity areas measured show development of systems, policy, and division of labor. Marginal Most capacity areas are reinforced with clear and documented policies, practices and systems. The organization is beginning to develop a track record of implementation and is starting to engage in participatory processes for change management. Expanding Capacity areas measured demonstrate a track record of achievement, accountability, participation throughout the organization as well as a mixture of flexibility and strong systems. Mature All capacity areas measured demonstrate sustainability as well as successful collaboration with a mixture of other organizations to ensure maximized impact. Partner Organizations Technical Area/s Regional coverage Capacity Stage & Rating (1 -5) Swaziland Business Coalition Against HIV & AIDS (SWABCHA) Prevention & HCT Hhohho, Manzini, Lubombo, Shiselweni Expanding (4) Swaziland National Network of People Living with HIV&AIDS (SWANNEPHA) Treatment Hhohho, Manzini, Lubombo, Shiselweni Marginal (3) The Salvation Army (TSA)* Care & Support & TB/HIV Hhohho, Manzini Expanding (4) Cabrini Ministries* Treatment, Paediatric Treatment, Care & Support & Paediatric Care & Support; OVC Lubombo Expanding (4) Good Shepherd Hospital (GSH) Treatment, Paediatric Treatment, Care & Support & Paediatric Care & Support Lubombo Expanding (4) Lutsango LwakaNgwane Gender / Sexual Prevention Manzini Emerging (2) Techno serve Swaziland OVC n/a Mature (5) Lutheran Development Services* (LDS) OVC Lubombo, Shiselweni, Manzini, Hhohho Marginal (3) Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 81 Coordinating Assembly of Non-Governmental Organizations (CANGO) Umbrella Grants Management (UGM) n/a Save the Children Swaziland (SCSWD)** OVC & Gender Hhohho, Manzini, Lubombo, Shiselweni Marginal (3) Voice of the Church (VOC)* & ** Prevention AB Hhohho, Manzini, Lubombo, Shiselweni Marginal (3) Bantwana Initiative** OVC & Gender Lubombo Expanding (4) Khulisa Umntfwana** Gender / Sexual Prevention Manzini Emerging (2) Cheshire Homes of Swaziland** Prevention, Care & Support Manzini, Lubombo, Hhohho Marginal (3) * Faith-based organization ** CANGO Sub-partner As PEPFAR/Swaziland’s largest OVC partner, PACT has made significant investments in the OVC response not just through sub-partners, but also through strategic national level efforts, such as the development of standards (OVC service standards, Early Learning Standards) as well as participation in national technical working groups and processes. 3. Evaluation Questions The final evaluation report produced under the resultant Task order shall 1) Use evidence to document relevance, effectiveness and sustainability of the interventions 2) Identify best practices and lessons learned; 3) Make recommendations for any future community-based interventions. The key evaluation questions to be addressed are: 9. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? 10. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? 11. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? 12. How does different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? 13. What approaches for delivering sustainable community services for OVC are working well? 14. To what extent has PACT strengthened efforts to improve the OVC response beyond sub-partner service delivery through activities related to developing of standards and other national processes? 15. What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? 16. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 82 4. Methodology To answer questions on the quality of implementation, USAID expects the contractors to apply a non￾experimental design approach with extensive use of quantitative and qualitative methods. These will include reviews of project documents and extensive use of routinely collected program data. The evaluators will also be expected to collect primary data, including key informant and focus group interviews and organizational capacity assessments of PACT’s sub-partners. Evaluators will be expected to conduct independent assessments of sub-partner organizational and technical capacity, and compare them to PACT’s assessment. The evaluation will also assess the relevance of the sub￾partner’s institutional strengthening plans developed by PACT. Prior to arriving in country and conducting field work, the team will review project documents and reports, to be made available by USAID/Swaziland. Upon award, but before fieldwork is conducted, the contractor will submit a detailed evaluation design, methodological framework, and implementation plan for review and approval by USAID. The following are illustrative data sources to be used by the evaluation team. Contractors are requested to propose other data sources and collection methods based on their understanding of the work to be done and proposed evaluation approach - Document review (project reports and project generated data; capacity building tools and institutional strengthening plans; relevant national documents). A detailed bibliography will be provided by USAID - Key Informant Interviews with PACT management and technical staff - Key Informant Interviews with sub-partner management and staff - Key Informant Interviews with volunteer caregivers affiliated to sub-partner programs - Key Informant Interviews with relevant government staff in line ministries (national/provincial level): NERCHA, National Children’s Coordinating Unit (NCCU)), Ministry of Health (MoH)/Swaziland National AIDS Program (SNAP), Directorate of Social Welfare (DSW), - Focus Group Interviews/discussions with beneficiaries of OVC programs (children and OVC caregivers) - Organizational capacity assessments The evaluators will conduct structured interviews with project staff, stakeholders and beneficiaries. To ensure that comparable information is collected during interviews, the team will develop interview guidelines for different groups of interviewees reflecting the evaluation questions. Field site visits to sub-partner head/field offices and implementation sites will be identified by USAID in consultation with PACT. Beneficiary interviews should concentrate on OVC programs with a focus on the Lubombo region, where the majority of PACT’s OVC partners operate. Sampling methods Given the largely qualitative nature of the primary data to be collected, it will be important for evaluators to propose sampling methods to minimize bias. E.g., systematic sampling of staff, beneficiaries, and sites must be done so as to get the full range of responses represented, not just those favored by the program implementers. Lists of staff, beneficiaries and sites will be provided to contractors to serve as a sample frame. Contractors shall develop innovative approaches to conduct this evaluation, using the attached program description for the project, the USAID Evaluations Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) and any other relevant information. It is expected that the contractor will discuss the relative strengths and limitations of the methodology proposed within the proposal. The methodology should take into account – and perhaps Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 83 independently assess, where possible – the quality of the routine monitoring data being collected by the projects. Additionally, the contractor should discuss data disaggregation (by gender and other categories) and gender considerations in the evaluation, particularly how to assess the differential effects of interventions on men and women. Data Analysis Quantitative data should be analyzed using frequencies, medians, and means and if appropriate, inferential statistics. Qualitative data should be analyzed according to the salience of responses and key themes that emerge from the data. Clean and coded qualitative and quantitative datasets will need to be shared with USAID upon project completion. Contractors are requested to complete the evaluation matrix below based on their proposed evaluation approach and data collection methods. Before data collection, the contractor in coordination with USAID will finalize the matrix and include it in the overall evaluation design and methodology plan. Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub-partners engage? To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub-partners to balance beneficiary coverage/targets with quality and dosage of services? How does different coverage of sub-partners (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services What approaches for delivering sustainable community services for OVC are working well? To what extent has PACT strengthened efforts to improve the OVC response beyond sub￾partner service delivery through activities related to developing of standards and other national processes? Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 84 Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods What progress has CANGO made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? Existing Data The project routinely collects data for PEPFAR standard indicators on Prevention, Care, OVC, Testing and Counseling, Treatment, Strategic Information, Health Systems Strengthening and Gender on which semi-annual reports are submitted to PEPFAR. The table below provides a sample of PEPFAR indicators and targets for each sub-partner (for illustrative purposes, for each organization the indicator with the highest target is presented). Full sets of indicators and targets for each sub-partner will be made available to the evaluation team. The team will have access to partners’ M&E data systems. Sample PEPFAR indicators collected by PACT sub-partners: Partner Indicator FY13 Target Lutheran Development Service Number of individuals who were provided with a minimum of one SUPPORT CARE service 9,426 Lutsango Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 1,800 TSA Number of individuals who received Testing and Counseling services for HIV and received their test results 888 SWANNEPHA Number of service outlets providing HIV-related CLINICAL care 4,230 SWABCHA Number of Most at Risk Populations (MARP) reached with individual and/or small group level interventions that are based on evidence and or meet the minimum standards 8,000 CABRINI Number of adults and children who were provided with a minimum of one care service (de-duplicated total of the CLINICAL CARE and SUPPORT CARE targets). 2,500 VOC Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 7,600 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 85 Khulisa Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 2,000 Bantwana Number of individuals who were provided with a minimum of one SUPPORT CARE service 8,024 CHESHIRE Number of individuals who were provided with a minimum of one SUPPORT CARE service 450 SCSWD Number of individuals who were provided with a minimum of one SUPPORT CARE service 18,587 GSH Number of HIV+ adults and children receiving a minimum of one CLINICAL service 3,300 PACT also collects baseline and routine data on 14 custom indicators related to organizational capacity building of its sub-partners according to PACT’s tools. These indicators and FY13 targets are presented below: PACT Performance Measurement Plan Indicator FY13 Target Overall Partner Capacity PACT 1: Percentage of partners which improve by at least 30% annually (or reach Level 4) on PACT’s Consolidated Organizational Capacity scale 75% PACT 2: Percentage of partners who report satisfaction with the support they receive from PACT 60% PACT 3: Number of Global Fund Recipients provided with technical assistance (mentoring, training) to implement HIV and AIDS programs (disaggregated by type of Technical assistance) No target set Governance and Leadership Capacity and Sustainability PACT 4: Percentage of partners which improve by at least 30% annually (or reach Level 4) on PACT’s Governance and Leadership Capacity and Sustainability scale (disaggregated by type of technical assistance and by source of technical assistance [i.e. PACT or Support Partner/s]) 75% Technical Capacity PACT 5: Percentage of partners which improve by at least 30% annually (or reach Level 4) on PACT’s HIV & AIDS Technical Capacity scale (disaggregated by type of technical assistance and by source of technical assistance [i.e. PACT or Support Partner/s]) 75% PACT 6: Percentage of partners who attain at least 85% of annual PEPFAR targets (disaggregated by technical area) 75% MER Capacity PACT 7: Percentage of partners which improve by at least 30% annually (or reach Level 4) on PACT’s MER Capacity scale (disaggregated by type of technical assistance and by source of technical assistance [i.e. PACT or Support Partner/s]) 75% PACT 8: Percentage of partner organizations passing annual PACT data quality audits (annual update) 85% Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 86 Financial Management Capacity PACT 9: Percentage of partners which improve by at least 30% annually (or reach Level 4) on PACT’s Management Capacity scale (disaggregated by type of technical assistance and by source of technical assistance [i.e. PACT or Support Partner/s]) 75% PACT 10: Amount of PEPFAR funding received by Implementing Partners through PACT managed grants (in USD) 2,364,998 PACT 11: Percentage of partners who submit accurate financial reports the first time each month 80% PACT 12: Implementing partner pipelines and burn-rates 90% PACT Grants Management PACT 13: Percentage of approvable financial reports from Implementing partners that PACT has paid within 10 working days 95% PACT 14: Percentage of implementing partners whose grant agreements are active by 1 October of each year 95% The following documents will be available for review:  Cooperative Agreement  Performance Measurement Plan (PMP)  Annual and semi-annual progress reports  PACT work plans  Capacity assessment tools and results  Institutional Capacity Building Plans for sub-partners  Site visit reports  Obligation Matrix  Data Verification reports  USAID DQA reports  Baseline and end line reports for WORTH savings groups  OVC Quality Service Standards, and baseline and endline reports from standards piloting phase  Sub-partner documents o Program descriptions o Grant agreements o Quarterly, Semi Annual and Annual Reports o MER Plans o Annual work plans and budgets o Organizational capacity assessments [End of Section C - Statement of Work] Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 87 ANNEX II: EVALUATION METHODS AND LIMITATIONS This Performance Evaluation utilized a non-experimental design that excluded a rigorously-defined counterfactual or comparison group. However, the evaluation team incorporated before-after comparisons to determine changes in project activities, outputs and outcomes over time. The team employed a mixed methods approach (collection of both qualitative and quantitative data) to collect both primary and secondary data and to ensure that the eight evaluation questions were answered comprehensively. The conceptual framework for the evaluation involved examining PACT capacity building processes (inputs), internal organizational changes among the sub-partners (outputs), and changes in the sub￾partners’ delivery of services (outcomes). Assessment of outputs and outcomes focused on elements of capacity (systems/policy, organizational, individual) as recently defined by PEPFAR’s Capacity Building Framework6. Examining these elements of capacity and capacity building will allow the evaluation team to determine the Effectiveness (in achieving outcomes, meeting standards), Efficiency (in delivering services and increasing reach using a minimum of inputs), Relevance (in engaging target populations and promoting learning), and Sustainability (mobilizing resources, increasing legitimacy) of REACH interventions. Conceptual Framework for Evaluation of Capacity Building Methodology The various methods employed in this Performance Evaluation included: 6 PEPFAR Capacity Building and Strengthening Framework. 2012. http://www.pepfar.gov/documents/organization/197182.pdf Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 88 a) Document and data review; b) Key informant interviews (KIIs) with PACT managers and staff; a selection of the nine sampled sub-partner managers and staff; and relevant stakeholders, i.e. Swaziland National AIDS Programme (SNAP), National Emergency Response Council on HIV and AIDS (NERCHA), National Children’s Coordination Unit (NCCU), Ministry of Education & Training, UNICEF and USAID; c) Focus Group Discussions (FGDs) with male and female beneficiaries, parents and caregivers affiliated to OVC sub-partner programs in 10 sites; and FGDs with the selection of the eight sampled sub-partner managers and staff plus CANGO; d) Technical Capacity Assessments with the eight relevant sub-partners; e) Confidential online survey of staff and managers for all 17 sub-partners. The design of the KII, FGD, and the online survey tools was guided by the eight evaluation questions. See Annex III for the evaluation tools used by the team. Document and Data Review Key to the evaluation was the comprehensive review of project documents and reports for the project and the 17 sub-partners. Document review provided the team with key background and contextual information and helped identify any data gaps. The evaluation team used existing documents along with the results of fieldwork to clarify the project’s underlying theory of change, specifically the outputs that were directly attributable to project activities, as well as the outcomes toward which the project contributed (but for which it is not solely responsible). The key documents and data that were reviewed are listed in Annex IV. Key Informant Interviews (KIIs) In-depth, semi-structured, key informant interviews were conducted to obtain perspectives on: the quality of the capacity building processes, the extent to which the desired system/ organizational/ individual changes have occurred, and the identification of gaps in sub-partner needs. The KIIs incorporated questions styled around Most Significant Change to capture the extent of change since the inception of the project. The KIIs were conducted with various staff within the sample of 8 sub-partner organizations and CANGO (CEO or head of organization, Finance Manager, M&E Manager; and Programme Manager). In addition, KIIs were conducted with representatives from:  USAID/Swaziland managers  PACT/Swaziland managers  Department of Social Welfare (DSW)  National Children’s Coordination Unit (NCCU)  MOH/AIDS programme (SNAP)  NERCHA  UNICEF Focus Group Discussions (FGDs) Focus group discussions were conducted with the selection of the eight sampled sub-partner managers and staff plus CANGO. Separate focus group discussions (FGDs) with OVC beneficiaries Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 89 ages 12-18, with caregivers, and with parents were conducted in ten sites where OVC services are delivered by the four relevant sub-partner organizations. All the FGDs utilized a semi-structured guide to ensure comparable information was collected at each site. The FGDs focused on their perspectives of the quality and extent of OVC service delivery, programme management, and any gaps identified. Prior to conducting the FGDs on OVC services, a Performance Scorecard was completed by each FGD participant on the satisfaction with and frequency (i.e. dose) of OVC service delivery categories. The scorecard was refined to ensure adequate quantitative indicators of the OVC services to reflect the range of respondents’ views. As an incentive for participation, the team provided parent and care giver focus group discussants with a R30 air time voucher and OVC beneficiaries with school supplies of comparable value. Technical Capacity Assessments At the head offices of the eight sub-partners delivering HIV/AIDS services the relevant Technical Capacity Assessments were administered to the group selected for the KIIs. The use of this tool allowed triangulation with the sub-partners’ previous Technical Capacity Assessment scores around the quality of service delivery and any changes since the Technical Capacity Assessments were administered in prior years. Confidential Online Survey An online survey collected feedback from the project’s 16 sub-partners and CANGO around their satisfaction with the quality and utility of the project’s organizational and technical capacity building efforts and the effects on organizational functioning and services delivery. Sampling Framework There are 17 organizations (i.e. sub-partners) affiliated with the PACT REACH project, including 4 that “graduated” in September 2013. A subset of eight (8) organizations were selected for in-depth interviews and external technical capacity assessments plus CANGO. Four (4) criteria were used for selecting the sample are as follows: Criterion No. Org meeting criteria Result Names of Organizations Selected CANGO 1 CANGO is included in the sample 1. CANGO Organization offers OVC services 4 All 4 organization offering OVC are included in the sample 2. Bantwana Initiative 3. Cabrini Ministries 4. Lutheran Development Service (LDS) 5. Save the Children Swaziland (SCSWD) A networking organization 2 Both the networking organizations are included in the sample 6. Swaziland Business Coalition on Health & AIDS (SWABCHA) 7. Swaziland National Network of People Living with HIV&AIDS (SWANNEPHA) Number of beneficiaries reached as of Sept 2013 2 Of the remaining organizations, the final two have a high number of beneficiaries and so were selected. 8. Khulisa Umntfwana 9. Voice of the Church (VOC) Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 90 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 91 These 9 organizations have varied length of experience with the project as shown below: Sampled Organizations’ Length of experience with REACH project Year started with REACH No. Organizations in Sample 2006 2 2007 0 2008 2 2009 2 2010 0 2011 2 2012 1 TOTAL 9 All 17 of organizations as well as several “graduated” organizations were invited to answer the online survey to obtain their views on the capacity building support provided by PACT. Response Rates Response Rates in the Evaluation Data Collection Method Target Actual Response Rate Key Informant Interviews 45 51 115% Focus Group Discussions 30 39 130% Technical Capacity Assessments 8 8 100% Confidential Online Survey 63 35 56% TOTAL 146 133 100% (avg.) The online survey which was sent to the organizations elicited 35 responses, or a 56% response rate, across present and past sub-partners of PACT as follows: Category of Respondents Targeted Actual Response Rate CEO, MD, or head of organization 16 7 44% Financial Manager 14 6 43% M&E Officer 13 9 69% Programme Officer 13 6 46% Other 7 7 100% TOTAL 63 35 56% Data Analysis The methodology described above generated a mix of qualitative and quantitative data which were analyzed using methods appropriate to each. Some of the techniques which the evaluation team used to analyze the data are briefly described below.  Theme analysis, which involved: o Viewing the data several times as a whole o Identifying patterns and themes o Reorganizing the data (e.g. coding the data according to the themes identified).  Triangulation: o Cross-checking the data in order to increase the confidence in the findings. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 92 o Use of multiple data sources (PACT and sub-partner managers and staff as well as beneficiaries) o Use of multiple data collection methods (document review, Technical Capacity Assessments, KIIs, online survey and focus group discussion).  Descriptive Statistics including graphs: o Trend Analysis o Frequencies o Percentages o Composite Scores where appropriate Methodology Summary Mapped against Evaluation Questions Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods 1. To what extent has PACT made progress towards effectively building the organizational capacity of its sub-partners? What have been the factors behind observed progress or lack of progress towards organizational capacity building? Changes in PACT OCA scores over time for all sub￾partners determined prior to the start of field work Separate from the fieldwork team and prior to instrument design, Data from existing OCA scores for each partner will be plotted to identify trends in capacity scores over time and to identify pace of progress. Based on these analyses, additional KII questions around the pace of progress will be developed for each sub-partner organization in the sample. Plotting of times series data from previous OCA scores Current OCA status assessed by the fieldwork without seeing PACT’s OCA scores in order to avoid bias and increase rigor OCAs administered by Khulisa evaluation team in the field Comparison of evaluation team’s assessment classification with those of PACT’s Most significant change in organizational capacity and factors behind the change Document review: Theme analysis KIIs with sub-partner managers and staff and PACT managers and staff Online survey questions on organizational capacity Frequency analysis of responses Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 93 Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods 2. To what extent has PACT made progress towards effectively building the technical capacity of its sub-partners (prevention, treatment/care, OVC)? Is PACT’s technical backstopping capacity adequate for the different technical areas in which sub￾partners engage? Changes in technical capacity scores over time will be used to identify slow progress prior to instrument design. Based on the results, KIIs will be customized to interrogate the reasons for slow progress per sampled sub￾partner. Separate from the fieldwork team and prior to instrument design, Data from existing OCA scores for each partner will be plotted to identify trends in capacity scores over time and to identify pace of progress. Based on these analyses, additional KII questions around the pace of progress will be developed for each sub-partner organization in the sample. Plotting capacity assessment scores over time Current technical capacity score PACT Technical Capacity Assessment Tool administered by Khulisa team in the field Comparison of evaluation team’s assessment classification with those of PACT’s Most significant change in technical or programming capacity and factors behind the change. Document review: Theme analysis KIIs with sub-partner managers and staff and PACT managers and staff Online survey questions on technical capacity support. The questions will be guided by the results of the trend analysis conducted on the PACT’s OCA scores over time. Frequency analysis of responses 3. To what extent has sub-partner service delivery improved to date under PACT’s support in terms of quality and targeting? How has PACT supported sub￾partners to balance beneficiary coverage/ targets with quality and dosage of services? Sufficiency of targeting Document review Graph trends over time Data review: service data Analyze sufficiency of targeting submitted to PACT Compliance of technical programme to PEPFAR guidelines (proxy for quality) PACT Technical Capacity Assessment Tool administered by Khulisa team Calculate average scores against key service delivery areas Document review Theme Analysis related to service delivery improvements in terms of quality and targeting KIIs with sub-partner program managers and staff Feedback and scores around balance of targets with quality/dosage KIIs with sub-partner managers and staff and PACT managers and staff Theme analysis related to service delivery improvements in terms of quality and targeting FGDs with volunteer caregivers affiliated with sub￾partner programmes 4. How does different coverage of sub￾partners Evidence of differences in coverage PACT Technical Capacity Assessment Tool administered by Khulisa team comparisons in technical scores between sub-partners working nationally vs defined geographic areas Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 94 Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods (national versus defined geographical areas) impact on the quality and dosage of services? How has PACT support helped sub-partners to engage and align with national strategies and systems, and to refer to government services? Evidence of alignment with national strategies and systems KIIs with (i) sub-partner managers and staff, (ii) PACT managers and staff, and (ii) national stakeholders. . Theme analysis Perceptions around the balance of coverage with quality/dosage FGDs Volunteer Caregivers 5. What approaches for delivering sustainable community services for OVC are working well? Feedback and satisfaction scores around community services for OVC FGDs w/Beneficiary and Volunteer Caregivers Theme analysis on approaches; documentation of different approaches A composite score for the sampled OVC programmes will be calculated based on the scores from the beneficiary FGD scorecards, and the technical assessment results conducted by Khulisa. KIIs with sub-partner managers FGD Performance Scores for OVC indicators Frequencies of scores, and theme analysis of reasons behind the scores. Document Review Theme Analysis Compliance of technical programme to PEPFAR guidelines (proxy for quality) PACT Technical Capacity Assessment Tool administered by Khulisa team Calculate average scores against key service delivery areas Document review Theme Analysis 6. To what extent has PACT strengthened efforts to improve the OVC response beyond sub￾partner service delivery through activities related to developing of standards and other national processes? Evidence of other capacity building activities at the national level. KIIs with PACT, NERCHA, MOH/SNAP NCCU, DSW, and others as appropriate Theme Analysis Document review Verification of KII responses. 7. What progress has CANGO CANGO’s current OCA status and Data review: Existing OCA scores for CANGO Graphing capacity assessment trends over time Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 95 Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods made towards developing its institutional capacity to become an Umbrella Grant Mechanism? How relevant is PACT’s Institutional Capacity Building Plan (ISP), and what progress has CANGO made on its ISP? changes in OCA scores over time OCA administered by Khulisa evaluation team Most significant change in organizational capacity and factors behind the change. Document review Theme analysis KIIs with CANGO managers and staff and PACT managers and staff Online survey questions on organizational capacity Frequency analysis of responses 8. What are the strengths and weaknesses of PACT’s approach to engage with civil society organizations that were recipients (or slated to be recipients) of Global Fund resources? How did PACT engage with the national AIDS coordination body NERCHA in order to strengthen civil society? Evidence of other capacity building activities of civil society Global Fund recipients (or potential recipients). KIIs with PACT, NERCHA, sub￾partners, and others as appropriate Theme Analysis Fit of PACT capacity building approach to Global Fund capacity requirements. Document review:  PACT plans, tools, reports  NERCHA documents  GF capacity requirements for PRs and SRs Comparison of PACT capacity building approach (methods and tools) to Global Fund capacity requirements:  Programme Governance  Programme management  Sub-recipient Management  Financial Management  Risk Management  Procurement and Supply Management  Monitoring & Evaluation Most significant change in organizational capacity and factors behind the change KIIs with PACT, NERCHA, MOH/SNAP, NCCU, DSW, and others as appropriate Theme analysis Limitations By their nature, qualitative evaluations of policies and programme implementation allow examination of the broader context, rather than specific measurements of compliance to policy elements or programmatic guidelines. This evaluation’s extremely broad scope (1 primary grantee focusing on organizational development, 17 sub-partner organizations implementing a wide range of HIV/AIDS and OVC interventions, and 8 research questions and sub-questions totaling 14 questions) resulted in lengthy interviews and focus group discussions with respondents who often had time constraints. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 96 Consequently, many respondents gave only cursory information around the issues that we sought to explore more deeply. Potential bias was possible from all respondents, including selection bias from the OVC beneficiaries, caregivers and parents/guardians as the team was not able to randomly select participants due to the need for service providers to contact potential participants in advance of the date the FGD was held and the time-constrained field work. Self-selection bias also was possible from those who agreed to participate. Thus there may be an overrepresentation of positive responses about OVC service delivery due to the lack of random sampling. There also could be an overrepresentation of positive responses about PACT’s capacity building due to potential interest in receiving further support from PACT, USAID, PEPFAR or the U.S. Government in the future. The team tried to mitigate this type of selection bias through the anonymous online survey and confidential one-on-one key informant interviews. Recall bias also was possible by participants due to inaccurate memory about activities and implementers. Results from a sample of KIIs and beneficiary FGDs may give an indication of capacity changes but not necessarily attribute those changes to PACT’s support. For example, a number of the organizations interviewed received capacity building from other organizations and individuals outside PACT. Moreover, FGDs and KIIs will not really allow for feedback on the change since programme inception, because all the participants were not with the programme since the beginning of PACT’s support and some sub-partners have not been operating in the same area or implementing the same services since 2010. Much of the data from previous years of assessments by PACT of its sub-partners was made accessible, but it was not possible to implement PACT’s organizational capacity and technical capacity assessment tools in the same manner as they were implemented by PACT due to time constraints. Consequently, the quantitative data collected using the PACT assessment tools is sparse compared to the data available for time series analysis. Also, increases and decreases in capacity development between 2013 and 2014 could not be objectively verified by the team. Moreover, there was no control or comparison group as a counterfactual or to attribute changes in capacity to PACT’s capacity building efforts. Also, the online survey data results representation was limited by the number of respondents to the online survey. The time available for research did not allow independent observation of service delivery. Thus independent verification of quality and dosage was not possible. In addition, “dosage” and “exposure” vary as the sub-partners implement different services, some sub-partners implement many different services, and some have received more PACT support than others. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 97 ANNEX III: DATA COLLECTION INSTRUMENTS Key Informant Interview Guide Name of Person Interviewed Name of Organization Position in Organization No. Years Working in Organization Gender of Respondent Male Female 1. What are you working on with PACT? 2. What is working best that PACT is doing? What is working well and why? What is not working well and why? 3. Do you know how PACT has helped to build technical HIV AIDS capacity? Please explain Are you aware of the methods PACT uses for its capacity building? e.g. training matched relevance, time and level of need of participants. Do you think their capacity building is efficient and effective? Explain. 4. Do you know what role does PACT play after capacity building? 5. Do you know of any specific challenges that have hindered capacity building approach or delivery by PACT? Please explain 6. What has been the most significant change in individuals, at the organization level and at the systems level as a result of PACT’s capacity building? 7. Do you feel PACT staff are skilled enough to provide the range of technical and organization capacity building skills that their sub-grantees need? 8. Do you believe that PACT provides a high-quality capacity building service? 9. How does PACT’s capacity building support sub-grantees’ targeting? 10. As far as you know, is PACT’s capacity-building work aligned to national policies and strategies? Please explain. Does PACT assist your organization with national policies and strategies and if so in what way? 11. Has PACT worked to strengthen the development of policy, or national standards setting, for service delivery, or anything else? Explain. 12. Do you know if the type, range or depth of services changed based on PACT’s support of its sub-grantees? If so, how? 13. Do you know of PACT’s work with its sub-grantees in a network/consortium and/or in a Technical Working Group? Please explain. Do you work with PACT sub-grantees as part of a network? If so, how? 14. Do you know about PACT’s work with its sub-grantees to address their sustainability (consider financial, organizational, human resources)? Please explain. Do you help PACT sub-grantees with their sustainability (finance, organizational, human resources)? If so, how? 15. Do you have any suggestions for PACT’s capacity building in the next two years? 16. Are you aware of PACT’s capacity building and strengthening of CANGO to deliver and manage services to sub-grantees? For PEPFAR? For Global Fund? Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 98 Technical Capacity Assessment Tool (Technical CAT) NAME OF ORGANIZATION: DATE OF ASSESSMENT: NAME OF ASSESSOR: STAFF INTERVIEWED: Rating scale: N/A Not Applicable X Insufficient information available to assess 1 Nothing in place/ needs urgent attention 2 Needs substantial attention 3 Needs some improvements 4 Needs some minor adjustment but without urgency 5 No need for improvements: systems adequately implemented & utilized A. PROGRAMME MANAGEMENT 1 Planning and Design Score Comments a Organization has a clear understanding of linkage of programs implemented to the organization strategy b Organization has clear engagement or participation processes for communities in planning and implementation of programs. c Organization has clear annual implementation plans, annual budgets and reporting structures. d Organization facilitates review meetings periodically to monitor progress of implementation e Organization uses volunteers with a clear volunteer management policy to drive programs f Program managers, program officers and community facilitators have a shared understanding of the program goals, objectives, and design model. g Organization identifies, maps and collaborate with program stakeholders h Organization use own qualitative and quantitative data and secondary data for evidence based program strategies. i Organization strategy clearly define the target population(s) by age, sex and geographical location j Organization has a documented referral system for complimentary support services offered by other organizations k Organization has designed a system to receive clients’ feedback on referral services l Organization meets regularly with the support service providers to review the referral system 2 Service delivery capacity Score Comments a Organization has infrastructure/facilities to implement the program b Organization size and structure is appropriate to implement the proposed program as evidenced by asset and inventory records c Organization has previous experience in implementing the type of program proposed. d Organization has a quality assurance system that supports delivery of services of consistent quality and content. e Organization has sufficient human resources with appropriate skills and experience to implement the program or can demonstrate ability to recruit and Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 99 A. PROGRAMME MANAGEMENT bring staff without impacting program timeliness. f Organization implementation plan demonstrates relevance to the beneficiary. g Organization implementation plan includes an exit strategy. h Organization developed a mechanism to regularly collect feedback on the effectiveness of the program from the target audience 3 Skills training for service delivery Score Comments a Organization has well defined and documented training curriculum / syllabus for community facilitators/ volunteers b Organization has a pool of trained personnel who can train implementers using the training manual / curriculum c Trainers are qualified in both the technical area(s) they present and effective training methods d Organization consistently train and refresher train all implementers including facilitators/ volunteers/ peer educators. 4 Community facilitators / volunteers /peer educators management Score Comments a Organization has established a sound support and supervision system for all community facilitators/ volunteers/ peer educators b Organization has well defined motivation and retention strategy for community facilitators/ volunteers/ peer educators c Organization has well defined and documented code of conduct, ethics agreement and other internal rules for community facilitators/ volunteers/ peer educators to promote role model behaviors. d One-to-one visits or meetings with community facilitators/volunteers/peer educators are scheduled to answer their questions and observe them at work. e Organization developed a checklist to guide supervisor support visits f Organization evaluates performance of community facilitators / volunteers/ peer educators and feedback to them at least once a year to identify strengths or needs for further support B. SEXUAL PREVENTION 1 Social and Behaviour Change Communication strategy (SBC Score Comments a Organization identifies and promotes specific skills and behavioral changes( abstinence, delayed sex, fidelity, reduced MCP etc. ) for specific target audience (disaggregated by age and gender) b Organization’s prevention strategy targets specific individual risk factors (e.g. unprotected sex, having multiple sexual partners at the same time, having monogamous partners but changing partners often, drug or alcohol abuse, Poor STI treatment seeking behavior, inconsistent use of condoms) c Organization’s prevention strategy targets specific societal risk factors (e.g. migration, sex to prove manhood or fertility, gender discrimination, sexual abuse, lack of employment opportunities, traveling or working away from home d Organization uses SBCC theories/models and evidence based information to plan and design HIV prevention interventions. e Organization selection of target audience is influenced by common drivers of HIV in Swaziland. (MCP, early sexual debut, intergenerational sexual relationships, high mobility, poverty, untreated STI, low levels of MC-HTC-consistent condom use, &, GBV) f Organization set clear social or behavioral change targets for specific population groups during planning and design g Organization follow national condom storage standards h Organization train and refresher rain all personnel who distribute condoms i Organization demonstrate the use of condoms before distribution of any Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 100 B. SEXUAL PREVENTION condoms to any audience j Organization has established links with facilities that provide Male Circumcision services and regularly refer men for this service C HIV TESTING AND COUNSELING 1 Pre and Post-test support services Score Comments a Organization selection of target population is influenced by common drivers of HIV in Swaziland. (MCP, early sexual debut, intergenerational sexual relationships, high mobility, poverty, untreated STI, low levels of MC-HTC￾consistent condom use, &, GBV) b Organization promote couple testing and counselling c Organization identify and promote specific attitudinal changes that support disclosure of one’s HIV status ( partner testing, supportive counseling d Organization developed strategy to support clients who access HTC services (support groups, post-test clubs, support against GBV, supportive counseling, referral services) e Organization uses multiple communication channels to reach and sensitize the target audience on HTC. (e.g. road shows, interpersonal communication, door to door, group approaches, dialogues, mass media, special events campaigns etc.) 2 Quality control and quality assurance Score Comments a Organization consistently and accurately collect and record lab samples for quality assurance b Organization ensures all test kits in use are within the expiry date c Quality assurance samples are effectively and efficiently transported to other labs as needed d Counseling is done in privacy and confidentiality of clients is preserved e Organization always has adequate supplies of reagents and test kits Organization screen for TB all clients who test Positive for HIV f Organization consistently and accurately collect and record lab samples for quality assurance 3 Waste management, safety and infection control Score Comments a Organization developed a comprehensive infection control strategy b Personal protective equipment is made available at all times ( gloves, masks,) c PEP (Post Exposure Prophylaxis) protocol is in place and all staff are aware of protocol d Organization follows national standards in the disposal of HTC sharps (needled, pricks et e Supplies for effective service delivery are available to every counselor (Disinfectants, puncture proof or metal container for disposal of sharp objects, gloves and other supplies for universal precautions) f Staff providing HTC services meet regularly to discuss patient care issues D IMPACT MITIGATION FOR ORPHANED AND VULNERABLE CHILDREN 1 Dimensions of quality for impact mitigation service provision Score Comments a Safety: Organization ensures safety of children by following the safety measures as recommended by the quality service standards for OVC. b Access: services for children are equitably distributed, accessible, affordable, considers children with disabilities, following access dimensions as recommended by the quality service standards c Effectiveness and efficiency: Organization’s desired results or outcomes are achieved in a timely manner, with resources needed minimised while the coverage is maximised Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 101 D IMPACT MITIGATION FOR ORPHANED AND VULNERABLE CHILDREN d Appropriateness and relevance: Organization adapts services to needs, circumstances, gender, age, disability, culture or socio-economic factors e Continuity: organization delivers on-going and consistent care, enabled by timely referrals, communication among providers, and networks with appropriate agencies. f Compassionate relationship: Organization establish trust, respect, confidentiality and responsiveness achieved through ethical practice, effective communication and appropriate interactions g Participation: Caregivers, communities and the children themselves participate in the design and delivery of services and in decision-making regarding their care. h Sustainability: Organization designs the service in a way that it could be maintained at the community level in terms of direction, management and procuring resources. 2 Impact mitigation service delivery Score Comments a Education Support: Organization’s education interventions ensure that barriers to children’s access to early childhood, primary and secondary education are identified and addressed and that the children’s progress is monitored b Psychosocial support: Organization’s psycho-social support interventions create a comprehensive supportive environment to meet the social, emotional, mental and spiritual needs of children and families by integrating PSS in all child-focused activities, by strengthening families’ and other caregivers’ understanding of PSS, and through offering appropriate services to children to strengthen their ability to cope and thrive c Protection: Organization’s protection services aim at reducing the risk of and protecting the child from physical, emotional or sexual abuse including violence, exploitation and neglect through strengthening community mechanisms for the identification, assessment, referral and monitoring of child protection services and empowering families and children d Shelter: Organization’s shelter services ensures that the child has a stable shelter that is adequate, dry, and safe, and that the child has at least one adult (age 18 or over) who provides consistent care, attention, and support. e Economic support: Organization’s economic strengthening services imparts livelihood skills to households that enable them to generate a modest income for attainment of basic needs. f Health: Organization’s health services support the prevention of illness, and access to treatment, health promotion and rehabilitation services for children through health education, trained and supervised community caregivers, structured referrals, and linkages with the health system g Food and nutrition: Organization’s food and nutrition services include an assessment of food and nutrition within households; the provision of nutrition education and/or agricultural improvements; and targeted nutritional support when necessary to ensure that children and their households are food secure and their nutritional needs are met on a sustained basis E PALLIATIVE HOME BASED CARE 1 General practice for quality palliative care Score Comments a Organization has documented rules to ensure confidentiality about the client’s case is kept among the professionals directly connected to clinical care. b Organization has documented rules about open and honest communication with client and family members c Client is always informed about all his or her conditions, options for treatment, benefits and disadvantages, probabilities of success and consequences of failure, as appropriate to his or her age. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 102 D IMPACT MITIGATION FOR ORPHANED AND VULNERABLE CHILDREN d Volunteers / caregivers receive adequate training to provide assistance and counselling that meets the expectations of the clients. e Program contributes to campaigns promoting client rights, anti-stigma and non￾discrimination against PLWHA f Organization has documented referral system with local/nearest health centre for provision of Art, treatment of OI, TB screening, CD4 assessment and other curative needs of the client. 2 Clinical / Physical Score Comments a Volunteers / nurses have standard operating procedures for home visits including assessment checklist, teaching topics, treatments, and referral information; volunteers carry memory aids (such as manuals, flowcharts, pictorial booklets) to remind them of procedures b Organization has strong links with a health centre for improved access to Pain and symptom treatment, referrals for ART & curative treatment c Organization teaches clients and family members where and when to access health care and support services d Organization teaches client and family members how to ease pain, support a healthy lifestyle, and prevent infection. e Clients are under close surveillance for active tuberculosis, and cases of active tuberculosis are aggressively treated, with support from the health centre. f Volunteers and staff doing home visits carry home-based care kits with all necessary tools for treatments and demonstration of techniques g Staff and volunteers/ caregivers are kept abreast of new developments in the field of palliative care, new resources for volunteers and clients, and new guidelines for palliative care programs through recent publications and participation in meetings, workshops and conferences 3 Social Score Comments a Client social needs are incorporated into care planning, care is adjusted to account for gaps in social support system. (e.g. child providing primary car b Gender specific issues such as male involvement in care, detection of domestic violence are assessed c Organization always searches and share information about new treatments, assistance, and new opportunities with all clients d Assessment of emotional wellness of client and family is done at every visit e Organization offers bereavement and grief counseling to families f Organization actively seeks assistance and support for client’s children with local OVC programs. F TREATMENT 1 General practice for treatment services Score Comments a Organization is registered with MoH as a health centre for provision of ART, treatment of OI, TB screening, CD4 assessment and other curative needs of the client b Organization has documented referral system with local/nearest health centre for provision of ART, treatment of OI, TB screening, CD4 assessment and other curative needs of the client c Organization provides HIV testing and counseling services or has well documented referral links with an HIV testing and counseling service provider d Organization facilitates the screening of TB in all HIV positive clients under their care e Organization train and refresher train counsellors who provide adherence counselling to clients on ART Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 103 F TREATMENT f Organization links facility care to community care for clients on ART. g Organization has a well-documented system of provision of comprehensive pre￾ART services. (HTC, CPT, CD4 count and linkages to care, adherence counselling and support,) i Organization has developed and documented linkages with Pre-ART system (pre￾ART register, files) and comprehensive care package (HTC, CPT, CD4 count and linkages to car j Organization equips personnel and volunteers with adequate treatment literacy to train PLWHIV. (Personnel and caregivers feel knowledgeable about CD4 counts, ART, ARV side effects and answers to most frequently asked questions about treatment.) k Organization develop or make available treatment literacy literature, brochures, training aides for personnel and volunteers l Organization links nutritional needs to ART services for children and adults on ART/ TB medication. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 104 On line survey for all organizations Introduction: Thank you for agreeing to be part of this survey which should take no longer than 12 mins to complete. You are asked to be honest and to rate how you feel about capacity building services and support which have been offered to your organizations by PACT. In this way, PACT will get feedback on how they are operating and this will inform the way they will precede in the future. This survey is not about finding fault but about giving information back to PACT to enable them to develop new approaches if needs be. PACT provides capacity building and support to your organization Gender of Respondent 1. Male 2. Female Technical Services Offered by Organization 1. Prevention 2. HCT 3. Treatment 4. Care & Support 5. TB/HIV 6. OVC Please answer all the following questions, using the following scale: 1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree; N/A = not applicable No Question Scale 1.. PACT’s capacity building strengthens the leadership in my organization 1 2 3 4 5 n/a 2. PACT’s capacity building enables my organization to deliver higher quality services 1 2 3 4 5 n/a 3. PACT’s capacity building has been at a proper pace for our organization to apply and absorb 1 2 3 4 5 n/a 4. The rate of change in my organization is slow 1 2 3 4 5 n/a 5. PACT is highly effective in building technical HIV/AIDS capacity in my organization 1 2 3 4 5 n/a 6. PACT technical HIV/AIDS capacity building helps our organization deliver a greater range of high quality services to our beneficiaries 1 2 3 4 5 n/a 7. PACT’s capacity building strengthens my organization 1 2 3 4 5 n/a 8. PACT does not offer additional support after providing technical HIV/AIDS capacity building 1 2 3 4 5 n/a 9. PACT’s capacity building interventions improve the individual skills of our staff 1 2 3 4 5 n/a 10. PACT’s support is not sufficient to raise the quality of the services we provide 1 2 3 4 5 n/a 11. PACT assists my organization to work more effectively with government 1 2 3 4 5 n/a 12.. The frequency of our support to beneficiaries has increased as a result of PACT’s direct support to our organization 1 2 3 4 5 n/a 13. PACT ‘s support helps us to reach the appropriate beneficiaries 1 2 3 4 5 n/a 14. PACT’s assistance has helped us to expand the number of beneficiaries that we support 1 2 3 4 5 n/a 15. We are better able to network because of PACT’s capacity building services 1 2 3 4 5 n/a 16. Our organization is more sustainable because of PACT’s capacity building 1 2 3 4 5 n/a 17. PACT support has helped us aligned our services to government policy and guidelines 1 2 3 4 5 n/a 18. PACT’s capacity building responds to all our organizational needs. YES/PARTIALLY/NO If no/partial – please describe the areas that PACT has not been responsive to your needs. 19. PACT focuses its capacity building program on our organization as a whole, rather than the skills of individuals 1 2 3 4 5 n/a 20. We receive technical HIV AIDS capacity building from other organizations and individuals YES/NO If yes, the technical HIV AIDS capacity building is better than that delivered by PACT 1 2 3 4 5 n/a 21. We receive organizational capacity building from other organizations and individuals YES/NO 1 2 3 4 5 n/a Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 105 No Question Scale If yes, organizational capacity building is better than that delivered by PACT 22. Within our organization, we review the relevance and effectiveness of PACT’s capacity building activities 1 2 3 4 5 n/a 23. PACT has a process for receiving feedback on its capacity building and support activities YES/NO 1 2 3 4 5 n/a 24. If yes, we actively provide PACT with feedback 1 2 3 4 5 n/a 25. PACT staff are sufficiently skilled to support capacity building of our organization 1 2 3 4 5 n/a 26. PACT ‘s partnerships with national structures help us to improve our services 1 2 3 4 5 n/a 27. Our HIV AIDS response improved because of PACT’s capacity building 1 2 3 4 5 n/a 28. PACT’s capacity building and support has enabled us to better engage other institutions/private sector in our response to HIV AIDS 1 2 3 4 5 n/a 29. PACT helps us to establish new partnerships to sustain our work. 1 2 3 4 5 n/a 30. Please add below any other comments you wish to make about PACT’s capacity building. ___________________________________________________________________________ Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 106 Focus Group Discussion Tools for OVC projects PACT/Swaziland REACH EVALUATION 2014 OVC PROJECT FOCUS GROUP SCORE-CARD PART 1: Scorecard Introduction to respondents: Thank you for being part of this discussion group. The purpose of this discussion is to evaluate the work being done by PACT which funds the project implemented by ______. We are trying to assess what approaches for delivering sustainable community service for OVC are working well. For the first part of the session I will read out a statement and please mark the expression that corresponds with your feelings with an x. This is similar to the method employed during the recent elections. Community name : OVC Project name : Date: Type of Respondent: 1. OVC 2. Parent / Guardian 3. Programme Volunteer / Care giver Gender of Respondent 1. Male 2. Female OVC SERVICE INDICATOR EXCELLENT GOOD AVERAGE POOR VERY POOR Information on program 1. Manner of program introduction (i.e. was protocol observed) 2. Quality of program introduction (i.e. was info. easy to understand) 3. Linkages to other organizations that provide services 4. Frequency of visits from government extension officers* Targeting 5. Relevant children in community targeted by project 6. Relevant children reached by goods and services Health 7. Access to health facilities for OVC and their families 8. Time it takes to get medical assistance in community 9. Access to medication 10. Number of home visits by community care-givers 11. Sensitivity of healthcare professionals Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 107 OVC SERVICE INDICATOR EXCELLENT GOOD AVERAGE POOR VERY POOR 12. Quality of care for PLWHA Education 13. Provision of grants for OVC, including secondary and high school 14. Assistance with securing school placement for OVC 15. Educational supplies provided to OVC 16. Sufficiency of homework assistance 17. Access to education for children with special needs Child protection 18. Life-skills training for young people 19. Education of parents / guardians on Child rights issues 20. Education of traditional Authorities on child rights issues 21. Education of service providers (teachers / nurses etc.) on child rights issues 22. Support for community based care givers 23. Access to HIV testing for Child Abuse survivors 24. Access to post exposure prophylaxis for Child Abuse survivors 25. Sufficiency of counselling for survivors of Child Abuse 26. Legal assistance for survivors of Child Abuse 27. Protection in the community (safe rehabilitation) Other services 28. Sufficiency of grief counselling for OVC 29. Support of Property rights for OVC 30. Support of Marriage rights for girls 31. Support for registration of Birth; Marriages and Deaths 32. OVC access to legal family and personal documents 33. Provision of transport to access health and social services 34. Availability of caretakers to accompany children accessing services 35. Peer support groups such as youth clubs 36. Access to information Food and nutrition Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 108 OVC SERVICE INDICATOR EXCELLENT GOOD AVERAGE POOR VERY POOR 37. Sufficiency of food for OVC 38. Sufficiency of school / community gardens initiated 39. Sufficiency of agricultural inputs 40. Quality of shelter provided by for OVC 41. Quantity of blankets and / clothing provided to OVC Economic strengthening 42. Quality of training on economic strengthening 43. Frequency of training on economic strengthening 44. Savings and credit projects 45. Quality of entrepreneur training 46. Frequency of entrepreneur training PART 2: Discussion Guide Thank you for completing the first part of our evaluation. Now let us talk a briefly about changes that have taken place i.e. whether the situation has deteriorated or improved since the ___________ project begun. This is a conversation so there are no right or wrong answers only individual views. 1. What was the situation in this community before this project began? [Probe: Who decided that this project was needed? What are the criteria for being a recipient? Is the assistance getting to the right people in the community? 1a) Not asked to caregivers – Are the community based care-givers / committees the right people to carry out the project? [Probe: How would you characterise your communication with them? 1b) Asked to Caregivers only - How often do you get visits from the field office? [Probe: Are there staff members specifically dedicated to this project? How would you characterise your communication with them? ] 2. How does this project assist OVC and their families to access health facilities? [Probe: In which way does the project provide care for PLWHA? Has the project helped with access to medication for those who need it?] 3. Has this project had any impact on the ability of OVC to access food? [Probe: Describe how and where.] 4. What support does the project provide to school going children? [Probe: do they help get school places for OVC? Do they supply materials; do they help with homework?] 5. Which referral services does the project provide? [Probe: which Health and Social services? Give an example of those currently accessed. Are these provided on time and when needed? 6. How have OVC been assisted financially? [Probe: Describe the activities they are involved in under this project. Which work and why?] Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 109 7. On which issues have you been educated through this project? [Probe: Has this education affected the behaviour of people in the community in any way? Give examples.] 8. How has the level of vulnerability among children in the community changed since the project began? [Probe: What are the main reasons for this? What practices, if any, should be adopted or what should be adapted?] Thank you for your time. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 110 Sub-Grantee Focus Group Questions These questions are adapted from 3 OCA tools (OCAT, MER-OCAT and MCAT). Say to interviewees: Please note that this is not an evaluation of you as a sub-grantee but of Pact and Pact’s support to sub-grantees….thank you for agreeing to speak to us…… For interviewer, after each section, ask: How has PACT helped your organisation in this area, either with Capacity Building or Support or both? Governance and Leadership  Governance: Does the Governing Body provide strategic direction? Does it bring in broader perspectives into how the organisation set and delivers its developmental goals? Does it develop linkages for greater impact and help raise expected standards? (Probe for leadership)  Senior Management Is the executive leadership of this organisation dependent upon one person or is it shared among several team members? Are the organisation’s decision-making processes transparent and open? Does senior management delegate decision making to relevant staff as appropriate? Does senior management strengthen the capacity of the organisation and is this a consistent process? (Probe for leadership)  Vision and Mission Do the staff and primary stakeholders (including the Board) have a good understanding of the vision and mission? Can tell me what you vision and mission are? Are the goals and objectives you set in the organisation carried through to the activities and the services offered to the beneficiaries? Are there times when this does not happen? 2. Management Practices  Organizational Structure and Culture Does your organisation have a clearly defined management structure and lines of authority and responsibilities which are understood by the staff? Does the organisational culture reflect the stated values, shared norms and principles that you follow? Do you believe that the organisational culture promotes excellence and commitment to high standards? Please give examples. Is there a culture of reflection and learning in your organization? Do your review your programs focusing on the lessons you have learned? Please give examples. Are there clear mechanisms in your organization for communication and sharing information, both vertically and horizontally? Explain.  Planning Does your organisation have a current and workable strategic plan which is in line with its vision and mission? When was this plan written and for what period? Do your program implementation plans reflect strategic objectives and the capacity of your organization to deliver on these objectives? Do you adjust your plans as a result of monitoring and evaluation processes? If yes, how often?  Human Resource Development Are staff motivated to acquire new knowledge and skills? Is there is a mechanism in place for sharing learning? Are there opportunities for staff to practice new skills and approaches? Do Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 111 they transfer new skills and knowledge to their colleagues? Is training and mentoring of staff a priority within the organisation? Give some examples. External Relations and Partnerships  NGOs’ Collaboration Please give a few examples of the beneficiary referrals for services your organization makes to other organizations/government?  Relationship with Government Partners Does your organization have communication linkages and does it work collaboratively with all of the relevant government agencies?  Relationship with donors and the private sector How is your organisation funded? Does it have a diversified funding base? Who are your donors? 4. Sustainability  Institutional Sustainability Does the organization have the capacity to adapt its structure according to changing needs? In your organization is succession planning taken seriously including nurturing leadership in middle and lower management? If yes, when did you start thinking about succession planning?  Financial Sustainability Does your organization have the ability to sustain its programs through a diversified funding base and adding new donors? How will this happen? Has your organisation developed linkages and partnerships with government and local businesses for financial sustainability? Please give some examples. 5. Internal Controls Are relevant staff trained in budgeting, finance and grants management through PACT’s capacity building programme? Explain. 6. Project Management How does your organization support program design and review, including setting targets? Are sub-grants closely monitored to ensure achievement of program objectives? Does this include site visits and detailed regular program reports? Is there adequate technical capacity to support program implementation? Does your organisation report to donors on time? How often do your report? Are your staff numbers adequate for program implementation and support operations? Is there a high turnover of staff? 7. Performance monitoring, evaluation and learning Is M&E built into your organization’s strategic plan/project documents/ Does your organization have input, output, outcomes, and impact program indicators representing different levels of result? Probe. Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 112 Does your organization have established systems and tools for data collection, collation and analysis? Describe your data quality management plan? Does your organization have established data verification and validation processes? 8. Program Management and Decision Making How do you use monitoring and evaluation information to influence program design, planning and implementation? Please give examples? 9. General How do you use your ISP? Is it yourself for you? What have been your best experiences with Pact? What have been your worst experiences or biggest challenge with PACT? ANNEX IV: SOURCES OF INFORMATION Documents Reviewed Cabrini Ministries Program Report 2012-2013 Capacity ISP Template Consolidated Institutional Strengthening Plans (ISPs) for Bantwana, Cabrini, CANGO, Khulisa Umntfwana, Lutheran Development Service, Save the Children Swaziland, SWABCHA, SWANNEPHA, Voice of the Church, 2011, 2012, 2013. Data Verification Form for Semi & Annual Reporting Data Verification Protocol Data Verification System Assessment Sheet Extended National Strategic Framework 2014-2018 (eNSF) GEM Scale Baseline Report Grants Capacity Assessment Results Summary Implementation Plans for Bantwana, Cabrini, CANGO, Khulisa Umntfwana, Lutheran Development Service, Save the Children Swaziland, SWABCHA, SWANNEPHA, Voice of the Church, 2011, 2012, 2013. Khulisa Umntfwana Parenting Manual Management Control Assessment Tool (MCAT) MCAT Results: REACH Partners, 2011 & 2012 MERL Capacity Assessment Results Summary Monitoring, Evaluation and Reporting Capacity Assessment Tool (MER-CAT) OD Capacity Assessment Results Summary OD Roadmap Assessment Tool Organizational Capacity Assessment Tool (OCAT) Organizational Development Monitoring Plan Organizational Performance Index Data Collection Form Overall Capacity Assessment Results: FY12, FY13 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 113 PACT Annual Program Reports 2010, 2011, 2012, 2013. PACT Annual Workplans, 2010, 2011, 2012, 2013 PACT DQA January 2011 PACT Organizational Performance Index (OPI) PACT Partner Information spreadsheets (3 versions) PACT Semi-annual Program Reports, 2010, 2011, 2012, 2013 PACT Swaziland fact sheet PACT Swaziland Mobile Technologies fact sheet PACT Swaziland Site Locations by Partner PACT Swaziland: Building Local Promise presentation Programs and Technical Capacity Assessments Results Summary, FY11, FY12, FY13 REACH Performance Monitoring Plan 2010-2015 Road to Wealth Book (WORTH) Standards for Quality Service Delivery for Orphans and Vulnerable Children in Swaziland, November 2012 Summary of Main Duties of PACT Staff Swaziland Civil Society Priorities Charter: An Advocacy Roadmap for the Global Fund to Fight AIDS, Tuberculosis and Malaria New Funding Model Swaziland Community REACH Award Swaziland Early Learning and Development Standards (SELDS), March 2013 Swaziland Ministry of Education & Training Teacher’s Handbook: Guidance &Counselling for Secondary Schools Swaziland Ministry of Education & Training Teacher’s Handbook: Guidance &Counselling for Secondary Schools: Activities Handbook Technical Capacity Assessment Tool WORTH Project End Line Survey Report, May 2013 WORTH Swaziland fact sheet People Interviewed Bantwana Initiative: Thulani Earnshaw, Executive Director Bhekithemba Mavuso, Programme Manager Thuli Mesengu, Financial Manager Mbuso Siwela, M&E Officer Cabrini Ministries: Diane Dalle Molle, Executive Director Bongani Khulamo, Health Programme Manager Ben Kickert, M&E Officer Barbara Staley, Deputy Director Nkosinathi Vilakati, Program Director, OVC CANGO: Emmanuel Ndlangamandla, Executive Director Kayise Nkambule, Programme Manager Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 114 Nompumelelo Phakathi, M&E Manager Gcebile Tsabedze, Grants Manager Department of Social Welfare: Moses Dlamini, Deputy Director Viera Hlatshway, Senior Social Worker Thandi Maziya, Director Khulisa Umntfwana: Precious Dlamini, Programme Manager Stella Lukhele, Director Zanele Masangane, Financial Manager Bhekithemba Shabangu, M&E Officer Lutheran Development Service: Sibusiso Dlamini, Executive Director Zandile Hlatshwayo, M&E Officer Gugu Khumalo, ECD Coordinator Nkuleko Nkhabela, Programme Manager Ministry of Education & Training: Gwendolyne Simelane Ministry of Health/Swaziland National AIDS Programme: Nomphilo Gwebu, Expert Client Coordinator Velephi Okello, National ART Coordinator National Children’s Coordination Unit: Makhosazana Mabuza, Psychosocial Support Specialist National Emergency Response Council on HIV/AIDS: Thembi Gama, Head of Programmes Nokwazi Mathabela, M&E Coordinator Nozipho Mkhatshwa, Impact Mitigation Coordinator PACT: Shombi Ellis, Organizational Development Manager Nonjabuliso Khumalo, Grants Manager Sam Kudhlande, MER Technical Advisor Choice Makufa, Technical Manager Nicole Miller, Country Director Ncamsile Tfwala, Programme Manager Save the Children Swaziland (SCSWD): Simon Khumalo, Programme Manager Mduduzi Makhabela, Financial Manager Dumisani Mnisi, Executive Director Nkosinathi Vilakati, Programme Director Swaziland Business Coalition on Health and AIDS (SWABCHA): Thobile Dlamini, Chief Executive Officer Wandile Dlamini, M&E Officer Neliswe Fakudze, Financial Manager Alice Tembe, Programme Manager Swaziland Network of People Living with HIV/AIDS (SWANNEPHA): Sipho Innocent Dlamini, Programme Manager Mthobisi Ncongwane, M&E Officer Thembi Nkambule, Executive Director Gcebile Simelane. Executive Director UNICEF: Kheto Dlamini, Child Protection Officer Muriel Mafico, Deputy Representative USAID: Wendy Benzerga, Deputy Mission Director Silke Felton, Impact Mitigation Specialist Natalie Kruse-Levy, Country Director Patrick Kunene, HIV/AIDS Program Manager/PEPFAR/U.S. Department of Defense Grace Masuku, Human and Institutional Capacity Development Specialist Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 115 Voice of the Church: Lomalangeni Dlamini, Programme Manager & M&E Officer Richard Dlamini, Executive Director Muzi Kunene, Financial Manager Abel Vilakati, Programme Manager Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 116 ANNEX VI: DISCLOSURE OF CONFLICTS OF INTEREST Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 117 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 118 Name Patricia Sullivan Title Organizational Development Consultant Organization Khulisa Management Services (Pty) Ltd Evaluation Position? ☐ Team Leader ☒ Team member Evaluation Award Number (contract or other instrument) Contract: AID-674-1-12-0002, Order No. AID 674-TO-14-0001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mid-term Evaluation of the RAPID and Effective Action Combating HIV & AIDS (REACH) Project in Swaziland I have real or potential conflicts of interest to disclose. ☐ Yes ☒ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 13 March 2014 Midterm Evaluation: Swaziland REACH Project Final Evaluation Report Page | 119 Name Sindi Nxumalo-Hleta Title OVC Consultant Organization Khulisa Management Services (Pty) Ltd Evaluation Position? ☐ Team Leader ☒ Team member Evaluation Award Number (contract or other instrument) Contract: AID-674-1-12-0002, Order No. AID 674-TO-14-0001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Mid-term Evaluation of the RAPID and Effective Action Combating HIV & AIDS (REACH) Project in Swaziland I have real or potential conflicts of interest to disclose. ☐ Yes ☒ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 13 March 2014 Commissioned by USAID/Southern Africa http://www.usaid.gov/southern-africa-regional U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523