May 2012 This report was produced for review by the United States Agency for International Development (USAID). It was prepared by Carl Henn, Deborah McSmith, Michele Tarsilla and Theresa Wilson through Social Impact. EVALUATION South Africa Umbrella Grants Management (UGM) Performance Evaluation USAID Southern Africa SOUTH AFRICA UMBRELLA GRANTS MANAGEMENT (UGM) PERFORMANCE EVALUATION May 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Carl Henn, Deborah McSmith, Michele Tarsilla and Theresa Wilson through Social Impact. SOUTH AFRICA UMBRELLA GRANTS MANAGEMENT (UGM) PERFORMANCE EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (http://dec.usaid.gov). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract No. RAN-I-00-09-00019. 1 South Africa Umbrella Grants Management (UGM) Performance Evaluation ACKNOWLEDGMENTS The Social Impact evaluation team for the USAID/South Africa Umbrella Grants Mechanism (UGM) Evaluation wishes to thank the USAID/South Africa colleagues who guided and supported our work in country, in particular Anita Sampson, Charles Mandivenyi, and Wendy Van Rensburg, as well as all the UGM activity managers who took time to enlighten us. We offer our sincere thanks to key South Africa government colleagues with whom we met in the Department of Social Development (at the national level) and the Department of Health (at the national, provincial, and district levels). We also wish to thank the UGM teams from Pact South Africa, FHI360, Right to Care, and FHI360 Capable Partners for making themselves available to speak with us on multiple occasions, and for substantial and much appreciated assistance in scheduling interview meetings with UGM sub￾partners. Our thanks go as well to all the UGM sub-partners and “graduated” UGM sub-partners (NPOs) we were fortunate to meet during this evaluation, as well as to the external technical support consultants we interviewed. And, last but not least, we thank the gracious staff at the Western Cape Gugulethu Midwife Obstetric Unit and Dr. Max Kroon, who helped us to understand the impact of the FHI360 CAP program on the quality of Prevention of Mother to Child Transmission (PMTCT) services and improved exclusive breastfeeding trends in the Gugulethu Township. SI Evaluation Team Carl Henn, Team Leader Michele Tarsilla, Evaluation Manager Deborah McSmith, Consultant Theresa Wilson, Consultant 2 South Africa Umbrella Grants Management (UGM) Performance Evaluation ACRONYMS AAHT Anglican AIDS and Health Care Trust AED Academy for Educational Development AIDS Acquired Immune Deficiency Syndrome AM Activity Manager (USAID) AOR Agreement Officer’s Representative (USAID) APS Annual Program Statement ART Anti Retro Viral Therapy ARV Anti Retro Viral BCC Behavior Change Communication BFHI Baby Friendly Hospital Initiative BOD Board of Directors C&C Communication and Coordination CAP FHI 360 Capable Partners Program CB Capacity Building CD Capacity Development CINDI Children in Distress Network CEO Chief Executive Officer CLMPU Childline Mpumalanga COB Chairman of the Board COP Chief of Party COHSASA Council of Health Service Accreditation of South Africa CPD Continuous Professional Development CWSA Child Welfare South Africa DDOH District Department of Health DNS Department of Nutrition Services DOE Department of Education DOH Department of Health DOA Department of Agriculture DSD Department of Social Development EB Exclusive Breastfeeding, Early Booking EC Eastern Cape ECD Early childhood development ED Executive Director ET Evaluation Team FHI360 Family Health International 360 FM Financial Management FP Family Planning FY Fiscal Year GM Grant Management GRIP Greater Nelspruit Rape Intervention Project HAART Highly Active Anti Retro Viral Therapy HCBC Home and Community Based Care HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus HPCA Hospice and Palliative Care Association HQ Headquarters IMS Information Management System INGO International NGO IYCF Infant and young child feeding KCA Key Competency Area 3 South Africa Umbrella Grants Management (UGM) Performance Evaluation KZN KwaZulu Natal LCCP Local CB Provider L&D Labor and Delivery MCWH Maternal Child Women’s Health M&E Monitoring and Evaluation MER Monitoring, Evaluation, and Reporting MERL Monitoring, Evaluation, Reporting, and Learning MMC Medical Male Circumcision MOU Memorandum of Understanding, Midwifery Obstetric Unit NACCW National Association of Child Care Workers NDOH National Department of Health NGO Non-Governmental Organization NIMART Nurse Initiated and Managed Anti-Retroviral Therapy NPO Non-Profit Organization NQA National Quality Authority NSP National Strategic Plan OD Organizational Development OVC Orphans and Vulnerable Children PA Patient Advocates PDOH Provincial Department of Health PEPFAR U.S. President's Emergency Plan for AIDS Relief PHC Primary Health Care PMTCT Prevention of Mother to Child Transmission RBM Results-Based Management RFA Request for Application (USAID) RFTOP Request for Task Order Proposal RTC Right to Care S2S South to South SAG South Africa Government SASSA South Africa Social Security Association SETA South African Sectoral, Education, and Training Authority SI Social Impact SP Sub-Partner SSP Sub-Sub-Partner TB Tuberculosis TC Training Center TOR Terms of reference TOT Training of Trainers TWG Technical Working Group UGM Umbrella Grants Management UGM COP UGM “Community of Practice” UGMP UGM Partner USAID United States Agency for International Development USAID/SA USAID/South Africa USG United States Government VCT Voluntary Counseling and Testing WC Western Cape WHO World Health Organization 4 South Africa Umbrella Grants Management (UGM) Performance Evaluation CONTENTS Executive Summary............................................................................................................................................................ 6 Purpose of the Umbrella Grant Mechanism (UGM) and Objective of the Evaluation...................... 6 Background and Introduction....................................................................................................................... 6 Evaluation Purpose and Limitations............................................................................................................ 7 Scope and Methodology................................................................................................................................. 7 Evaluation Conclusions and Recommendations........................................................................................ 8 Introduction.......................................................................................................................................................................13 Background ....................................................................................................................................................13 UGM Partner Background ..........................................................................................................................15 Leader with Associate Award Background ............................................................................................... 16 I. Evaluation Purpose, SOW Questions, Evaluation Design and Methodology..........................................18 Purpose of the Umbrella Grant Mechanism (UGM) Evaluation..........................................................18 Key Evaluation Questions........................................................................................................................... 18 II. Findings, Conclusions, And Recommendations............................................................................................ 22 Grant and Financial Management ..............................................................................................................23 Capacity Building (CB)................................................................................................................................. 30 Defining and Measuring Success (M&E) ..................................................................................................37 Cross-Cutting Thematic Areas....................................................................................................................40 UGM Assistance to the SAG: Pact and CAP........................................................................................... 47 Summary – Looking Ahead.........................................................................................................................50 Appendix A. Scope Of Work..........................................................................................................................................54 Appendix B. Methodology ..............................................................................................................................................58 Appendix C: SI Evaluation Report Standards of Evidence.......................................................................................62 Appendix D. Persons Contacted...................................................................................................................................63 Appendix E. Surveys/Questionnaires........................................................................................................................... 66 I. Online Survey .......................................................................................................................................67 UGM Questionnaire .....................................................................................................................................72 Sub-Partner Questionnaire .......................................................................................................................... 77 FHI 360 Capable Partners Questionnaires ............................................................................................... 87 DSD Questionnaire ....................................................................................................................................105 UGM Follow-Up Questionnaire ..............................................................................................................108 5 South Africa Umbrella Grants Management (UGM) Performance Evaluation Graduate Sub-Partners...............................................................................................................................112 Appendix F. Case Studies..............................................................................................................................................114 UGM Sub-Partner Case Study: Agri-Aids...............................................................................................115 UGM Sub-Partner Case Study: Anglican AIDS and Healthcare Trust (AAHT)..............................120 UGM Evaluation Case Study: Child Line Mpumalanga (CLMPU) ....................................................123 UGM Evaluation Case Study: C* (Governance and Leadership Case Study)...................................127 UGM Evaluation Case Study: Children in Distress Network (CINDI).............................................131 UGM Evaluation Case Study: DSD: UGM Case Study: External Technical Support through UGM134 UGM Evaluation Case Study: Foundation for Professional Development (FPD)..........................136 UGM Evaluation Case Study: Greater Rape Intervention Project (GRIP).......................................140 UGM Sub-Partner Case Study: Hospice Palliative Care Association .................................................144 UGM Sub-Partner Case Study: Humana People to People (HPP).....................................................149 UGM Sub-Partner Case Study: Kheth’Impilo (“Think Life”).............................................................154 UGM Sub-Partner Case Study: National Association of Child Care Workers (NACCW) .............157 UGM Sub-Partner Case Study: Nurturing Orphans of AIDS for Humanity (Noah)......................161 UGM Sub-Partner Case Study: South-to-South Partnership for Comprehensive Family HIV Care and Treatment Program.....................................................................................................................................165 UGM Evaluation Case Study: World Vision/South Africa (WV/SA)...............................................170 UGM Sub-Partner Case Study: Woz’obona (“Come and see”)...........................................................173 Appendix G. FHI 360 Capable Partners (CAP) Analysis.......................................................................................178 Appendix H. Umbrella Grant Mechanism (UGM) Summaries..............................................................................189 FHI 360.........................................................................................................................................................190 Pact ................................................................................................................................................................194 Right To Care...............................................................................................................................................198 Appendix I. Options for Future UGMs......................................................................................................................200 Appendix K. References................................................................................................................................................202 6 South Africa Umbrella Grants Management (UGM) Performance Evaluation EXECUTIVE SUMMARY In late 2006, USAID/SA (South Africa) announced that it would organize and run two concurrent, related procurement competitions in order to establish an umbrella grant management structure that would help the Mission to manage a large number of sub-partners and expand service delivery in response to the huge influx of PEPFAR funding for HIV/AIDS. The announcement of APS 674- 07-011 led to the launch of the UGM projects evaluated here. One competition was to select the UGM partners (UGMPs); through the other, USAID selected sub-partners (SPs) to assign to the UGMPs through which USAID/SA would channel funds and provide capacity building. The reason given by USAID/SA for selecting the SPs directly was to be able to manage the complex processes of coordinating and allocating PEPFAR funds annually across all PEPFAR program areas, in accordance with funding guidelines (e.g., 55% for treatment or 10% for OVC). USAID/SA indicated that it could not delegate the annual funding level decisions by program area to the UGMPs. The FHI360 Capable Partners (CAP) program, also included in this evaluation report at the request of USAID/SA, provided technical assistance (TA) to the South African Government (SAG) Directorate of Health (DOH). The purpose of CAP was to integrate infant and young child feeding guidelines and nutritional counseling into prevention of mother to child transmission (PMTCT) services. The project was technical, and trained clinicians as well as community caregivers; it did not provide or manage subgrants. PURPOSE OF THE UMBRELLA GRANT MECHANISM (UGM) AND OBJECTIVE OF THE EVALUATION The purpose of the USAID/South Africa (SA) Umbrella Grant Management (UGM) evaluation was to inform the design of the next UGM. The evaluation focused on strengths and weaknesses of three bilateral UGM partners (or UGMPs): FHI360, Pact, and Right to Care. It also focused on a fourth project (the FHI360 Capable Partners project) in a variety of domains. FHI360 Capable Partners, or CAP, was not a UGM, but rather a Leader with Associate Award under a global award. Therefore, throughout the report, UGMPs is used to refer to the first three UGMPs, and CAP is addressed separately, due to its different aims, methodologies and funding mechanism. BACKGROUND AND INTRODUCTION The UGMPs were useful to USAID/SA in two key ways. First, UGMPs provided USAID with a practical means to channel relatively moderate amounts of funding (in most cases) to a large number of local sub partners (SPs). This allowed USAID to expand the number of SPs, while restricting the burden of management on the mission. Second, the UGMPs provided capacity building and grant and financial management support to the SPs, and in so doing sought to strengthen their capacity as organizations. In some cases, larger SPs did well enough that they “graduated” and became prime partners of USAID. Examples of successful “graduates” include the Foundation for Professional Development (FPD) and the Hospice and Palliative Care Association (HPCA). 7 South Africa Umbrella Grants Management (UGM) Performance Evaluation CAP also focused on supporting SAG development of national PMTCT and nutrition curricula and training materials. Given limited funding and staff, CAP was impressive in providing training to large numbers of clinical sites, the materials it helped develop, and documented improvements in PMTCT services and positive behavioral changes among HIV positive mothers. Readers will nevertheless note that CAP had little in common, real or implied, with the UGMPs, and was simply included in the evaluation at the request of USAID in order to have a rapid assessment done. EVALUATION PURPOSE AND LIMITATIONS The UGMPs evaluation questions were developed to focus on past performance (Summative evaluation) and future design (Formative evaluation). Questions about past performance explored each UGMP’s reported strengths and weaknesses, and the extent to which they enhanced or inhibited UGMPs success or that of the SPs they supported. The second set of questions addressed what lessons could be learned from the UGMPs experiences to inform USAID /SA’s design of future UGM mechanisms. As the current mechanisms end in 2012, design inputs were needed in a timely fashion to support permit procurement activity to proceed. The evaluation team sought to provide multiple design options, based on performances and learning to date. The Request for Task Order Proposal (RFTOP) under which the evaluation was conducted called for an evaluation team rather than a procurement design team. The skill set of the evaluation team was well-matched to the retrospective analysis. For the prospective design aspect of the evaluation, a disclaimer should be made that the evaluation team was not privy to any procurement-sensitive information, and therefore had no inside knowledge as to which options or approaches USAID might prefer most, or be most likely to use or to reject. The options suggested in the report are informed, logical options based on the analysis done by the evaluation team, extrapolated from the performances and experiences observed by the team. SCOPE AND METHODOLOGY Scope In December, 2011, USAID/SA issued Request for Task Order Proposal (RFTOP) 674-12-00006 to evaluate the UGM mechanism. The purpose of the RFTOP was to complete an external evaluation of three Umbrella Grants Management mechanisms – Pact South Africa, FHI360, Right to Care, and one Leader with Associate award, FHI360 Capable Partners. The RFTOP was for a Firm, Fixed-Price Task Order. The purpose of the external evaluation was stated as follows: • Determine the extent to which the UGM has achieved its goals and objectives in grant making and capacity building; what has worked and what has not worked and why; • Identify challenges in the areas of grant management and capacity building; • Analyze UGM activities and recommend modifications or enhancements; • Assess impact and benefits of investments on local organization capacity building;, • Identify best practices to inform future design of this mechanism. 8 South Africa Umbrella Grants Management (UGM) Performance Evaluation Methodology Social Impact deployed a four person team to South Africa for five weeks in March-April 2012 to conduct the evaluation in five of South Africa’s nine provinces: Gauteng, KwaZulu Natal, Western Cape, Mpumalanga, and Limpopo. The evaluation design called for sixteen mini-case studies (three to five pages in length), incorporating findings from semi-structured interviews of UGMPs and SPs, as well as an online survey of all SPs. The team reviewed key program documents (budgets, work plans, M&E plans, activity reports), as well as external evaluations previously conducted both for UGMPs and SPs, where available. The team interviewed USAID Activity Managers (AM) and Agreement Officer’s Technical Representatives (AOTR), SAG Department of Social Development (DSD) and DOH stakeholders, external technical support consultants, and conducted an extreme case (strongest and weakest) sampling of SPs. The team conducted an initial briefing and a final debrief for the Mission Management team, and received excellent support and guidance from USAID/SA throughout the process. EVALUATION CONCLUSIONS AND RECOMMENDATIONS Grant and Financial Management, including Application Process and Pre-Award Assessment The evaluation team’s overall observation, based on documentation and interviews with UGM and USAID, is that all three UGMPs ensured compliance and accountability for US funds. There were few instances of noncompliance or nonperformance by the SPs managed by the UGMPs; in these limited cases, the evidence seen suggests that the UGMPs discovered the issues quickly and took prompt action. The SI team observed that minor flaws in financial management, governance, or leadership, if not addressed rapidly and pro-actively, have the potential to escalate to major flaws. Recommendations 1. USAID should allow UGMPs to provide annual inputs on SP performance, absorptive capacity, and management capacity to help determine funding levels and better manage pipelines. 2. USAID should establish a more fully transparent process for selecting SPs and assigning or re-assigning them to UGMPs. 3. USAID should spell out transparent UGM graduation and/or exit criteria for SPs. 4. USAID should ask SPs to complete CB self-assessments as part of their UGM applications, and consider their self-scores as part of the selection criteria. 5. USAID should ask for technical self-assessments on the programming side to determine if SPs have both organizational and programmatic merit. Self-scores could be verified in a pre￾award or post-award stage to ensure each SPs provided correct information, meets minimum 9 South Africa Umbrella Grants Management (UGM) Performance Evaluation eligibility requirements, and complies with USG rules. Using this information, USAID could consider what mix of pre-existing partner capacity it desires, and how many newer, smaller, lower capacity partners USAID and UGMPs can both manage. 6. There should be concrete plans to address needs identified and build capacity with explicit, timely milestones for the SPs to achieve, as well as clear, specific indicators to measure CB progress. The UGMPs and USAID should monitor progress toward milestones, verified against a clear action plan as a criterion for future funding. 7. It is suggested that USAID utilize a simple alert system, such as “yellow cards” or “red cards” which UGMPs could use to signal USAID about urgent concerns with the SPs and lead to a series of actions or processes to reduce the risk of further problems. Capacity Building (CB) or Technical Assistance (TA) provided by the UGMPs The UGMPs marshaled an impressive array of tools and methodologies to assess the initial capacities of SPs, identify gaps and weaknesses, and develop and carry out plans of action to build capacity in a variety of areas of organizational development. However, three weaknesses were identified with the use of assessment tools:  Administration of tools was not as systematic and frequent as it could have been;  UGMPs often did not provide enough feedback to SPs on self- assessments; and  The information collected did not always inform development of tailored CB plans. Given the large number of SPs interviewed, and the ample opportunity they had to criticize their UGMPs anonymously, the team found that the SPs appreciated the CB and TA they received, and many SPs said it helped them grow as organizations. Interviews and documents provided considerable anecdotal evidence that they had done so. However, there were differences between the UGMPs with respect to each one's approach to providing CB and TA. The SI team concludes that Pact UGM had a unique and noteworthy mechanism to recruit and fund external TA for SAG DSD and DOH, beyond the usual CB components provided by the other UGMPs. For the DSD, Pact provided external TA for the Child Care and Protection and HIV and AIDS Directorates. External TA provided to DSD at national level was intended to focus on or cascade to the provincial level. SAG Key Informants were clear that the external TA through Pact contributed to improvements in quality of services, through both capacity building trainings and development of guidelines and standards intended to capacitate social service professionals to provide quality service delivery. Recommendations 1. Organizational capacity assessment should include a SP self-assessment followed by a UGMPs assessment and a CB plan, followed by regular/annual re-assessments. The next UGM might intensify CB in all seven key capacity areas identified in the APS 2006 (financial management, organizational management and governance, program management, quality assurance, strategic information and reporting, and leadership and coordination of partner organizations implementing prevention activities). 10 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2. The UGM CB model should be a “blended” model based on principles of adult learning (self-reflected, experiential, and participatory) and encompass the three following activities: • Coaching: Intensive short-term CB focused on a specific area/topic, such as the financial and monitoring and evaluation requirements. • Mentoring: Long-term CB focused on more developmental related topics, such as governance, resource mobilization, and strategic use of information for decision-making. • Community learning events: Collective reflection characterized by sharing of experiences as well as dissemination of successful operational and M&E tools. Theory of Change, and Monitoring and Evaluation of Results Analysis of Theory of Change The Theory of Change implicit in the UGM included the assumptions that by providing CB and TA to sub-partners, the UGMPs would enable them to perform better as organizations, to deliver more and better services, and to reach higher “PEPFAR targets” than would otherwise have been possible. In other words, the UGM model presumed that local organizations would need CB and TA, without which they would not have been able to manage and account for USAID/PEPFAR funds, comply with US government rules and regulations, and reach numeric “targets” set by PEPFAR for the number of clients receiving services such as HIV/AIDS care and support, prevention, or treatment. It was difficult for the SI evaluation team to answer these fundamental questions from the available evidence, e.g., to what extent did the CB and TA from the UGMPs contribute to the success or failure of the SPs, either as organizations, or in terms of reaching intended PEPFAR targets? While some SPs succeeded to such an extent that they “graduated” (moved from being sub-partners to become “prime” or direct USAID partners), in some cases, other SPs had to be suspended or dropped after instances of noncompliance, poor accountability, or irresolvable crises of leadership and/or governance. In numeric terms, of the 16 SPs the team examined, far more SPs succeeded (15) than failed (1); as stated above, two succeeded in becoming prime partners. This success rate is to the credit of USAID and the UGMPs, as well as to the SPs themselves. It was perplexing that a few SP failures occurred in spite of mammoth, long-term efforts by the UGMPs to prepare SPs to succeed as organizations and as service delivery providers. In spite of the due diligence of USAID and the UGMPs, it proved to be impossible to prevent every single failure, and finally, the team was challenged to define to what extent CB and TA provided by UGMPs contributed to overall organizational or program quality. Neither the UGM nor the SPs incorporated detailed initial baselines on organizational performance or program quality, and neither UGMP nor SP M&E plans included detailed indicators on these topics. A more vexing question would be whether the SPs could have produced the same results and achieved the same targets without any investment in CB; if this were true, critics might ask why invest in CB at all? Since there were no baseline indicators, and no control groups, from which to collect this data during the life of the project, there was not a sufficient evidence base on which to draw any definitive conclusions. Therefore, it is impossible to draw direct correlations between CB and ultimate success of the SPs, although interviews strongly suggest that CB was seen as beneficial to most, if not all, SPs. 11 South Africa Umbrella Grants Management (UGM) Performance Evaluation Recommendations In the future, UGM designs should incorporate a more explicit causal model, and a reasonable number of indicators of organizational performance, program quality, and impact of capacity building. The aim should be to allow USAID and UGMPs to demonstrate the extent to which the CB and TA inputs translate into desired outputs and outcomes. USAID/SA should aim to determine how much CB and TA is required to achieve a given level of performance and productivity among partners. It seems wise in the future for USAID to anticipate crises and failures, to try to prevent them by all means, but always to be prepared to act quickly and decisively when and if the need arises. Way Forward - Design Considerations for the next UGM How should USAID proceed in terms of future UGMPs design in order to obtain the best results? USAID/SA has indicated that it intends to proceed with a UGM mechanism in the future, partly out of necessity to manage the large number of local sub-partners, and partly due to policy directives that call upon USAID to continue to increase local partnerships and build local capacity. However, when responding to these broad imperatives with another UGM, USAID/SA must confront a number of complex UGM design issues: One or multiple UGMs? Focus on technical expertise or organizational development? Organize the UGMs by geographic or program area? Match indigenous partners with indigenous UGM or with international partners? Focus on larger partners with better prospects to become primes, or support small NGOs? Consolidate For practical purposes, given that USAID/SA seeks to reduce its management burden, all logic suggests that USAID/SA should try to have the smallest number of UGMPs possible, in order to keep its management burden as low as possible. More UGMPs would require more USAID staff to manage them. The fewer UGMs, the fewer UGM offices, the less rent, the less senior staff, and the fewer vehicle fleets there will be, reducing, in all likelihood, total management costs. Multiple projects generate multiple local management structures, all at a price. Since the UGM is based on the notion of “pass through,” this means that the awardee(s) cannot impose the usual overhead or indirect cost rates on the total amount of direct funding. However, UGMs can take a reasonable percentage for administration, CB, and TA; therefore, fewer are better. However, it is up to USAID/SA to determine whether the minimum number of UGMs should be one or more. USAID/SA will have to take into consideration a host of policy directives, as well as to consider procurement issues to which the SI Evaluation Team was not privy, in making a decision. Customize and Localize CB and TA The evidence from all SPs interviewed indicated that more geographically-dispersed UGM staff and support would be appreciated by the SPs, as would more on-site CB and TA. USAID/SA needs to consider the benefit of allowing UGM to post staff in provincial or regional hubs, versus “hoteling” them in central offices, and requiring them to travel to the field to deliver CB and TA. A simple way 12 South Africa Umbrella Grants Management (UGM) Performance Evaluation to decide the matter is through competition: ask the applicants to propose the best cost and management models, and analyze the applications to determine the best value for government. One likely configuration for a future UGM would be a consortium model, combining the best of the CB and organizational development features of one or more UGMPs, with the technical expertise offered by other UGMP. Applicants can form consortia they feel are competitive based on the terms of the procurement. USAID/SA does not need to prescribe a particular model, though it might want to describe the desired qualities and capabilities needed for a consortium to win. Shift Implementation to SAG The model USAID/SA seeks should also strive to provide enhanced contracted CB and external TA support to the SAG, along the lines of the model developed by Pact, and should also seek to engage the private sector to a greater degree to leverage new resources, ideas and approaches. This would be consistent with policy statements that USAID/SA will be moving away from service delivery towards a TA model, in which SAG takes the lead in service delivery. It would also allow USAID/SA to tap the vast resources of the private sector, consistent with USAID Forward reforms. 13 South Africa Umbrella Grants Management (UGM) Performance Evaluation INTRODUCTION USAID South Africa is a partner in the United States Government PEPFAR program to prevent the spread of the HIV virus through testing and education and to provide clinical and social support to those already infected with the virus. PEPFAR provides substantial amounts of funding for this effort, while the USAID/SA mission is primarily responsible for organizing the implementation programs associated with the PEPFAR objectives. In 2011, USAID/SA issued a Request for Task Order Proposal under the Evaluation services IQC to conduct a process, or Performance Evaluation of the PEPFAR program, namely the Umbrella Grant Managers project. Social Impact Inc. won the award, fielding a 4 person team in April-May 2012 to conduct the evaluation. The SI team worked closely with USAID and during the process, additional themes and issues emerged which were not specifically included in the SOW, but which the SI team addressed. This report presents the team's findings, conclusions and recommendations for both SOW questions as well as emergent themes. The team's substantive draft has been reviewed and accepted by USAID/SA. SI's Quality Assurance process requires that a senior SI evaluation specialist review each report, and, working closely with the team leader, take all steps necessary to ensure that the final report submitted to USAID meets SI standards, as well as being responsive to the USAID clients instructions and comments. In this instance the ET prepared a report that was responsive to the USAID/SA mission's interests and needs, but in SI's corporate view, the final draft presumed too much knowledge of the UGM program to be understandable by other USAID officers and other readers outside of USAID/SA. The report underwent substantial revision by a senior SI advisor with the help of the SI team in an effort to produce a report that would be both understandable and credible to higher level and more diverse audiences. In this revised version, we have sought to bring to the fore additional background information, as well as the best evidence to bear in a transparent effort to address the questions posed in the original SOW, as well as those that emerged during the course of the evaluation. In this effort we were guided by Social Impact's Quality Assurance standards, themselves closely modeled after best practices found in USAID, DOS, MCC and American Evaluation Association's policies. The SI Standards statement may be found in Appendix C. BACKGROUND In late 2006, USAID/SA issued Annual Program Statement 674-07-011, or APS, calling for applications for proposals for a grant management and capacity building program using PEPFAR funds. USAID/SA stated that it would select two or more Umbrella Grant Manager Partners (UGMPs). The purposes of the UGM were stated as: • To facilitate further scale-up of HIV/AIDS services • To develop indigenous (implementation) capacity of local NGOs to deliver those services. The APS also stated that UGM would not implement project activities directly, but act as grant managers to manage and mentor local Sub-Partners and Sub-Sub Partners (SPs and SSPs) who 14 South Africa Umbrella Grants Management (UGM) Performance Evaluation would implement directly. USAID would allow the UGM partners (UGMPs) only a small percentage of funds for administrative costs, but would allow the UGMPs to set aside a “reasonable” percentage of funds for technical assistance (TA) and management support. The PEPFAR program categories for the UGM were listed as: • Prevention • Care • Treatment • Policy, Strategic Information (SI) and System Strengthening (SS) The APS gave as examples of the types of Capacity Building (CB) activities UGMP would provide to SPs: Strategic Planning; Registration; Financial Management; Human Resource Management; Resource Mobilization; Logistics Management; (Procurement of) Commodities and Equipment; and Facilities. The UGM would be required to provide timely, quality support to the SPs, and to comply with (USAID and PEPFAR) reporting requirements. The APS only required UGMPs to report on two PEPFAR indicators: 1. Number of local organizations provided TA for HIV-related services 2. Number of local organizations provided TA for HIV-related policy In a departure from the more common practice of UGM partners selecting their own sub-partners, usually subject to USAID approval, USAID clearly stated that it would select the sub-partners (SPs) for the UGM, and assign them to the successful applicant(s). USAID/SA also stated that it would select the SPs from among those currently working with PEPFAR funding in South Africa, or those USAID selected to do so in the future. (USAID/SA told the SI evaluation team this was due to its need to manage the annual allocations to SPs across PEPFAR program areas in respect of changing Congressional earmarks). Ultimately, three UGM agreements were awarded to: AED (which later became FHI360), Pact, and Right to Care (RTC) for a total life-of-project value of $348,303,920 across all the awards. A fourth award, a Leader with Associate or LWA, was made to FHI360 for a project called Capable Partners, for $9,445,000. All awards began 10/01/2007 and will end 09/30/2012. Details of the awards for the four projects evaluated are provided below. 15 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 1: Distribution of SP Responsibilities by UGM Partner UGM Partner # Sub-Partners (SPs) Budget Special Features Pact 30 $239, 562, 000 Provision of Technical Assistance to the Department of Social Development + Comprehensive package of organizational development and technical capacity building services available to SP FHI 360 Development 8 (3 other SP were closed out) $ 90, 741, 920 Right to Care (RCT) 3 (one more was closed out) $ 18, 000, 000 LWA-FHI 360 Capable Partners (CAP) No SP supported (not a UGM in the strict sense). It is mostly a provider of capacity building to SAG DoHealth $ 9, 445, 000 Capacity building in Prevention of Mother to Child Transmission (PMTCT) among Department of Health in 4 Provinces Total: $348, 303, 920 UGM PARTNER BACKGROUND Pact UGM Agreement value (ceiling): $239,562,000 Pact supports institutional capacity building, technical assistance and grants administration for both indigenous South African organizations and US NGOs that implement PEPFAR programs. Pact enhances local sub-partner capacity through the development and implementation of documented organizational systems and procedures and human capacity development at management and operational levels. Pact conducts initial assessments identifying key organizational strengths and weaknesses, and works with each sub-partner to develop and implement a tailored, phased capacity building agenda. Pact provides funding and assistance to over 30 total PEPFAR partners and sub partners in South Africa. FHI360 (formerly known as Academy for Educational Development (AED) UGM Agreement value (ceiling): $90,741,920 FHI360 supports institutional capacity building, technical assistance to, and grants administration for indigenous South African organizations that implement PEPFAR programs. FHI360 also aims to 16 South Africa Umbrella Grants Management (UGM) Performance Evaluation achieve: (i) increased sustainability of HIV prevention, care and treatment efforts and enhanced country ownership; (iii) increased graduation of indigenous NGOs from sub-funding roles to direct USG funding. FHI360 partners and sub-partners include local NGOs, FBOs, and CBOs selected through the Inter-Agency PEPFAR Annual Program Statement (APS). FHI 360 supports financial management, program management, quality assurance, strategic information (M&E) and reporting, and leadership and coordination of partner organizations implementing care and treatment activities. Right to Care UGM Agreement value (ceiling): $18,000,000 Right to Care (RTC)'s UGM project supports sub-partner organizations through financial oversight, project management, human capacity development, training, mentorship programs, program development, treatment expertise, and strategic planning in providing Adult HIV Care and Support (ACS) services. RTC disseminates policies and guidelines and provides quality assurance by sharing best practices and increased monitoring. LEADER WITH ASSOCIATE AWARD BACKGROUND FHI 360 Capable Partners (formerly known as Academy for Educational Development [AED]) Agreement value (South Africa Task Order): $9,445,000 FHI 360 Capable Partners (CAP) collaborates with PEPFAR-funded Prevention of Mother-To￾Child Transmission (PMTCT) partners to strengthen PMTCT services in four provinces (Free State, North West, Limpopo and Western Cape). The goal is to improve performance of selected PMTCT sites by promoting best practices. FHI 360 conducts PMTCT Training of Trainers (TOT) for auxiliary nurses and lay counselors, to equip them with appropriate knowledge and skills of PMTCT. With the provincial departments of health (PDOH), CAP provides TA to PMTCT facilities to improve service quality. CAP transfers skills to trainers to train providers, as well as to providers directly. The scope of work for the CAP TA includes: (a) pre-service and in-service training courses for auxiliary nurses and lay counselors; (b) clarifying expectations for newly trained staff and managers and strengthening supportive supervision (c) strengthening referral systems to enhance continuity of care; (d) improving referrals from PMTCT to Family Planning (FP) services; (e) training on couple counseling, and; (f) helping facilities include FP messages to increase uptake of FP. UGM Program Organization and Processes The UGMPs were useful to USAID/SA in two key ways. First, UGMPs provided USAID with a practical means to channel relatively moderate amounts of funding (in most cases) to a large number 17 South Africa Umbrella Grants Management (UGM) Performance Evaluation of local sub partners (SPs). This allowed USAID to expand the number of SPs, while restricting the burden of management on the mission. Second, the UGMPs provided capacity building and grant and financial management support to the SPs, and in so doing sought to strengthen their capacity as organizations. In some cases, larger SPs did well enough that they “graduated” and became prime partners of USAID. Examples of successful “graduates” include the Foundation for Professional Development (FPD) and the Hospice and Palliative Care Association (HPCA). CAP also focused on supporting SAG development of national PMTCT and nutrition curricula and training materials. Given limited funding and staff, CAP was impressive in providing training to large numbers of clinical sites, the materials it helped develop, and documented improvements in PMTCT services and positive behavioral changes among HIV positive mothers. Readers will nevertheless note that CAP had little in common, real or implied, with the UGMPs, and was simply included in the evaluation at the request of USAID in order to have a rapid assessment done. 18 South Africa Umbrella Grants Management (UGM) Performance Evaluation I. EVALUATION PURPOSE, SOW QUESTIONS, EVALUATION DESIGN AND METHODOLOGY PURPOSE OF THE UMBRELLA GRANT MECHANISM (UGM) EVALUATION The purpose of the USAID/South Africa (SA) Umbrella Grant Management (UGM) evaluation was to inform the design of the next UGM. The evaluation focused on strengths and weaknesses of three bilateral UGM partners (or UGMPs), FHI360, Pact, and Right to Care. It also focused on a fourth project (the FHI360 Capable Partners project) in a variety of domains. FHI360 Capable Partners, or CAP, was not a UGM, but rather a Leader with Associate Award under a global award. Therefore, throughout the report, UGMPs is used to refer to the first three UGMPs, and CAP is addressed separately, due to its different aims, methodologies and funding mechanism. KEY EVALUATION QUESTIONS As per the scope of work (Appendix A), the Social Impact (SI) evaluation team (ET) was tasked with addressing three sets of questions: (1) general evaluation questions, (2) capacity building-related questions, and (3) grants management-related questions. The USAID RFTOP SOW questions are as follows: General • What key features of each UGM mechanism, enhanced or prohibited successful implementation and achievement of the key program objectives? • What key lessons can be drawn from the successes and or failures should be considered for future program planning? Capacity Building • What key features of each UGM mechanism enhanced or prohibited capacity building efforts? • What were the key results, strengths and weaknesses and unanticipated results of the capacity development processes implemented by each UGM Mechanism under the grant management program? • What are the key lessons learned/promising practices that emerged from the UGM provision of capacity building support (e.g. in the areas of Monitoring and evaluation, finance, program management and implementation and organizational development)? • What indicators may be used to monitor a UGM’s capacity building work? Grants Management • What key elements in the internal management structure and systems of UGM mechanisms contributed (positively or negatively) to achievement or failure to achieve program results over the implementation period? • What key features of each UGM mechanism enhanced or prohibited grant management efforts? 19 South Africa Umbrella Grants Management (UGM) Performance Evaluation • What key elements and appropriate management structures have been most effective that may be incorporated into future grant management mechanisms? • What key lessons can be drawn from the successes and or failures that should be considered for future program planning? In agreement with USAID/South Africa, the team assessed the two following additional questions: • What key features of each UGM mechanism enhanced or prohibited successful implementation and achievement of the key program or project objectives, including achieving overall development aims? • What key lessons drawn from the successes and/or failures of the projects should be considered for future program planning? The SI evaluation team was able to address the evaluation questions above in a greater number of key areas than those initially identified in the RFTOP: These key areas are: (1) application and pre￾award assessment, (2) defining and monitoring success, (3) service delivery, (4) innovation, (5) sustainability, (6) governance, and (7) coordination and communication. Social Impact Team SI deployed a four-person team to conduct this evaluation in South Africa from March 10 to April 13, 2012. The four-person SI ET split into two mini-teams to maximize their ability to carry out the scope of work. The two mini-teams conducted data collection in five of the nine South Africa provinces (Gauteng, Kwa-Zulu Natal, Western Cape, Limpopo, and Mpumalanga,). The team completed its field work and presented the key elements of its findings, conclusions and recommendation prior to departure from South Africa. Evaluation Research Design This evaluation was a "Performance Evaluation" consistent with USAID Evaluation Policy (2011). The evaluation team worked closely with USAID throughout the evaluation, and was responsive to additional USAID suggestions. The evaluation adopted a non-experimental Mixed Method plus case study design for four main reasons. First, an experimental or quasi-experimental design might not have adequately captured the variety of organizational features (e.g., size, areas of intervention, years of activity, etc.) characterizing the SPs supported by the UGM mechanism. Second, an experimental design (e.g., a randomized, controlled trial) was not feasible, as the UGM mechanism had already been implemented for several years without either baseline data collection or establishment of a control group. Third, quasi-experimental design, for similar reasons, was not viable, either. And finally, the nature of the questions included in the evaluation requested by USAID/South Africa did not focus so much on impact, but rather on process-related issues and overall UGM effectiveness consistent with the basic character of a USAID Performance Evaluation. 20 South Africa Umbrella Grants Management (UGM) Performance Evaluation Data Collection Methodology The evaluation design and methodology developed by the SI evaluation team was used to assess the comparative performance of the three UGMs. USAID/SA had chosen to use UGMs to fund NGO delivery of HIV services across all South African provinces. During the first phase, by capitalizing on the information already available, e.g., number of grants recipients, disbursement rates, type of activities funded through the grants, and the number of people targeted by such activities, the ET assessed key differences across the three UGMs, with emphasis on implementation and organizational development processes, i.e., the extent to which UGMs contributed to increased local NGO (both UGM sub-partners and sub-sub-partners), absorption capacity, and quality in delivery. The evaluation featured a mixed method design including: • Program document review: The ET reviewed UGMP Annual Reports as well as other key program documents (annual budgets, work plans, M&E plans) available for each of the 16 SP organizations visited during the fieldwork portion of the evaluation. • Semi-structured questionnaires: The SI team developed a variety of semi-structured questionnaires (see Appendix E) for the following categories of UGM key stakeholders (a different version of the questionnaire was administered depending on the type and level or respondents):1 - UGM USAID/South Africa Activity Mangers - UGM Partners (UGMP) - Pact, FHI360, Right to Care - UGM Partner (FHI360 Capable Partners) - UGM Sub-Partners (SP) and Sub-Sub Partners (SSP) - DOH (National, provincial, and district) - DSD (National) – UGM External Technical Assistance (TA) Providers – Follow-up with UGMPs (Pact, FHI360, Right to Care, and FHI 360 Capable Partners • Mini case studies (16): The SI team developed mini-case studies of approximately three￾four pages in length on sixteen SPs and SSPs identified through two different purposive sampling strategies: extreme case sampling strategy (i.e., the best and least performing SPs and SSPs, etc.) and critical case sampling (SPs and SSPs that stood out either because of their 1 The SI team prepared an online survey instrument for SPs to answer per the original data collection design,, but USAID/SA chose to manage the survey directly. Data was not used in the formation of this report. 21 South Africa Umbrella Grants Management (UGM) Performance Evaluation innovative approach or unique organizational features) (See Appendix F for the individual case studies). • Interviews: The ET then visited the 16 sub-partners and used structured and semi￾structured questionnaires to interview over 100 people among SP and SSP staff, Department of Health (DOH), and Department of Social Development (DSD) officials (Appendix E). In addition, the SI team also conducted extensive interviews and met frequently with USAID staff, and met UGM partners (UGMP) and SP/SSP throughout the duration of the evaluation. Data Analysis Due to the nature of the case study design, the evaluation mainly relied on qualitative data analysis. Based on an in-depth review of UGM-related documentation, and the open-ended interviews held with UGMPs during the first week in country, a number of key UGM topics (“emerging themes”) was identified. By providing a snapshot of the experience of each SP in relation to emerging themes the mini-case studies allowed the team to generalize findings. This was also facilitated by a four-day intensive session of group data analysis. The SI team also triangulated findings using multiple sources (program documents, semi-structured interviews with several stakeholders at multiple levels, and key informant interviews both within and outside of USAID). Limitation of Bias The use of triangulation (derivation of findings from the convergence of multiple methods,) helped to limit researcher bias. Subjective judgments To further limit subjectivity, the SI team established a “member-check-in” mechanism within the team (regular, in-depth calls when the team was apart or face-to-face meetings when the team was together) and held regular weekly meetings with the client, USAID. Each evaluation team member as well as USAID/South Africa was given the opportunity to review the preliminary findings and to ask that those not clearly supported by evidence be revised. Conflict of interest Overall, the SI team did not have any specific conflict of interest vis-à-vis the UGM program being evaluated. However, as one of the evaluation team members (the local South African Orphans and Vulnerable Children [OVC] specialist) had carried out a three-week consulting assignment for one of the UGMPs (Pact), Social Impact made sure that this person was not involved in either the data collection or the data analysis related to the UGMP in question, as attested by the specific clause included in the consultant’s contract. 22 South Africa Umbrella Grants Management (UGM) Performance Evaluation II. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Organization Rather than organize the main body of the evaluation by close adherence to the original SOW's three main question sets under the headings: General, Capacity Building, Grants Management, and Lessons Learned, the SI Team, in consultation with the USAID AOTR, uses a different set of topics to organize the evidentiary and analytic structure of its report. The discussion of these topics includes answers to the questions posed in the SOW. The balance of the report is organized by the following thematic topics. Each numbered section or sub-section will present Findings, Conclusions, Recommendations, and for most sections, "Best Practices," and Lessons Learned. The final sections will address "options" for USAID going forward with the next round of UGM grants. The remainder of this report is organized by 6 main thematic sections: Grant and Financial Management The Application and Pre-award Process Financial Management (FM) and Grant Management (GM) Capacity Building Defining and Measuring Success (M and E) Cross Cutting theme and issues: Program Service Delivery Sustainability Communication and Coordination Innovation Assistance to the SAG Special Case Study; Pact CB assistance to SAG/DSD CAP "Looking Ahead": options for the next round of USAID support for the PEPFAR HIV/AIDs support program. 23 South Africa Umbrella Grants Management (UGM) Performance Evaluation GRANT AND FINANCIAL MANAGEMENT Application and Pre-Award Assessment Findings In 2006, USAID issued two Requests for Application (RFAs), one for UGMPs and one for SPs, selected the awardees, determined SP funding levels, and assigned them to UGMPs. This created a triangular relationship of USAID-UGMP-SP, which presented coordination challenges. Most SPs competed to win awards. Some SPs were in effect “grand-fathered in” due to prior funding under one or another mechanism, which had been or was being discontinued. USAID used the UGM to channel the funds to complete the projects. After completing the initial RFA process, USAID continued to decide SP assignments and funding annually, based on PEPFAR targets to be reached in each program area and assessments of each SP’s ability to reach them. USAID reduced its management burden with the UGMPs, but by soliciting applications for SPs and directly selecting them, USAID also took on some additional responsibilities. All UGMPs expressed concern that they were not consulted about SP funding levels or the number of SPs assigned to them, either initially or annually thereafter. They also indicated interest in assisting USAID to determine future funding levels by sharing information about SP absorptive capacity, performance, capacity-building (CB) progress, burn rates, etc. Sub-partners would prefer to be involved in deciding which UGMPs would manage their grant and help them build their capacity. Four of the larger SPs felt they should already be direct, USAID partners. All UGMPs conducted some form of SP pre-assessment before disbursing grant funds. The pre￾assessments determined if SPs met minimum, financial-management prerequisites, and identified CB priority areas related to compliance. Based on the evidence, the pre-assessment processes for Pact and FHI360 appeared to be the most comprehensive, and the best documented for later reference. Pact employed the most collaborative approach. RTC conducted initial self-assessments. Pre-assessments resulted in development of plans to build the capacity of SPs based on identified areas of need. However, progress on these plans varied by UGMPs and SPs, and did not appear to affect decisions about subsequent funding. Each UGMP also conducted reassessments annually, at minimum. The reassessments measured progress towards CB targets, identified any new concerns, and formed the basis for further annual plans. Based on interviews, SPs indicated that they preferred voluntary rather than compulsory pre￾assessments and reassessments, and preferred assessments that were presented as a means to help partners improve, rather than to ensure compliance with donor requirements. The SPs understood and accepted that compliance was mandatory, but nevertheless preferred an approach that was presented as collaborative in nature rather than as a policing function. Conclusions 24 South Africa Umbrella Grants Management (UGM) Performance Evaluation Pre-assessment and reassessment, in theory, help both the UGMPs and SPs identify needs, track progress on earlier findings, and plan for the year ahead. However, in practice, it is not clear that progress influenced USAID decisions about funding the SPs. Furthermore, despite the intuitive expectation that UGM capacity building would improve SP performance, there is no clear, consistent correlation between SP pre-assessment, reassessment, and: • Capacity building results compared to assessments; • SP ability to reach their PEPFAR targets or the quality of services provided; and • Timely detection of compliance problems or of crises of governance or leadership. This could be because few, if any, indicators were tracked to provide correlations, but it also speaks to the highly variable nature of organizational performance. The main value of the pre-assessment seemed to lie in determining initial eligibility for funding and partnership with USAID. It may or may not spot potential compliance problems or leadership issues. While regular assessment has certain a theoretical value and intrinsic appeal, the team questioned its ultimate impact, absent a set of clear indicators to establish success or failure. Recommendations 1. Strengthen the Award. Budget allocation and Grant Management process All SPs going forward should apply and compete for funding to ensure complete transparency and equal opportunity. It should no longer be necessary going forward to grand-father in SPs. USAID should allow UGMPs to provide annual inputs on SP performance, absorptive capacity, and management capacity to help determine funding levels and better manage pipelines. USAID should seek to establish an even more transparent process for selecting SPs and assigning or re-assigning them to UGMPs, perhaps establishing and sharing its criteria for doing so. USAID should also spell out transparent UGM “graduation” and/or exit criteria for SPs. Here, “graduation” could have two meanings: To SPs, it means they succeeded in managing substantial amounts of resources, and qualified for funding as USAID direct partners; for USAID, “graduation” suggests partners acquire other donors and are weaned off of USAID funding. “Exit” is a neutral term which means either that they were found to no longer need UGMP support or have value to USAID, or perhaps that they did not qualify to continue with a UGMP due to poor performance. 2. Strengthen the Capacity Building assessment and action process In the future, USAID should ask applicants to complete CB self-assessments as part of their UGM applications and consider these self-score as part of the selection criteria. In addition, USAID should ask for technical self-assessments on the programming side to determine if SPs have both organizational and programmatic merit. Self-scores should be verified in a 25 South Africa Umbrella Grants Management (UGM) Performance Evaluation pre-award or post-award stage to ensure each SP provided correct information, met minimum eligibility requirements, and complied with USG rules. Using this information, USAID should consider what mix of pre-existing partner capacity it desires, and how many newer, smaller, lower-capacity partners USAID and UGMPs can both manage. Future pre-assessments and reassessments should result in concrete plans to address needs identified and build capacity with explicit, timely milestones, as well as clear, specific indicators to measure progress on CB. The UGMPs and USAID should consider progress toward milestones, verified against a clear action plan monitored by the UGMPs, as a criterion for increasing future funding levels. 3. Develop an "Early Warning" system for tracking SPs compliance and performance. Another key issue is what to do about minor “early warning signs” identified in assessments. The SI team observed that minor flaws in financial management, governance, or leadership, if not addressed rapidly and proactively, have the potential to escalate to major flaws. USAID should ask UGMPs to develop a simple alert system, such as “yellow cards” or “red cards.” UGMPs should to signal USAID about urgent concerns with SPs, to initiate a series of rapid actions or processes to contain the matter, and to reduce the risk of further problems. 4. Integrate and Accommodate UGMP to emerging "USAID Forward Initiative" Another set of considerations derives from review of the USAID Forward initiative, which will require USAID missions to partner with more local organizations and private-sector entities, and to utilize more fixed-price contracts. USAID/SA should consider how this applies to UGMPs. For example, USAID should require UGMPs to describe how they would build the capacity of various types of partners working under the relevant types of contracts or awards. There would need to be different types of indicators and progress measurements for private-sector entities, which must operate at a profit in order to sustain themselves in the long term. 5. Align UGM SP plans with SAG. The PEPFAR Partnership Framework, drafted by PEPFAR South Africa and the South African government (SAG), requires that PEPFAR SA continue to align its priorities more closely with those of the SAG in the future. Therefore, future UGMPs should try to align SP strategic plans with those of the SAG. USAID could require partners to synchronize with SAG planning and budget cycles, i.e., rather than allow them to develop their budgets and plans “off-cycle” totally independent and irrespective of SAG processes. Lessons Learned For USAID and other donors The evidence examined shows that Umbrella Grant Management (UGM) mechanisms are a useful management tool for USAID Missions with limited staff and large portfolios, numerous local partners, and high levels of funding for which they must produce results and are held accountable. 26 South Africa Umbrella Grants Management (UGM) Performance Evaluation The models employed by USAID/SA are promising ones worthy of replication, though the design elements and the modes of implementation should be tailored to the needs of each situation. However, UGMs also present a set of management challenges for donors. Among these is the need to determine how much of the scarce resources available should be allocated to capacity building versus to direct service delivery, or to building long term sustainability, or to other potential uses, and on what basis such decisions should be made, and against which data/evidence. Provision of CB and TA through UGMPs to local SPs that might otherwise not be able to receive or manage donor funding is a laudable aim. However, donors must consider carefully how to determine its ultimate value. The application and pre-assessment processes provide two key opportunities for donors to learn about local SPs and guide their capacity building and pursuit of results. Donors should require thoughtful design and implementation of UGM CB efforts, and pay close attention to what their money buys. For Prospective or actual UGM partners or sub-partners NGOs and other organizations participating in, or considering a UGMP or SP role, may want to consider the findings and conclusions of this evaluation in their application planning and design phase, as well as during implementation. This information presents them with a chance to identify best practices, as well as an opportunity to avoid earlier mistakes and to improve performance. All SPs, but local SPs in particular are encouraged to view pre- and re-assessment as opportunities to establish a clear baseline and measure progress against it, and to adhere closely to action plans and timetables developed. There are many potential pitfalls for local SPs when managing USAID resources for the first time. Despite the challenges and demands of managing USAID funding, they should take every occasion to strengthen their systems and build their capacity for future success, whether receiving USAID or other donor funding. Financial Management (FM) and Grant Management (GM) Key Findings There was a cascade of grants made from USAID to UGMPs, and from UGMPs to SPs. In terms of USAID awards to UGMPs, no major delays or difficulties were reported. Though initial UGMP selections were reported to be quick, several months passed between notification of award to completion of negotiations and disbursement of funds. In the future, reducing these delays will accelerate start-up. Most UGMP awards to SPs took just a few weeks, provided that SPs met minimum eligibility standards. The SI evaluation team observed that there are essential tensions and trade-offs between (1) ensuring SP compliance and accountability as a primary goal and (2) balancing compliance concerns with implementation needs, allowing UGMPs and SPs more time to focus on performance and producing results. Up to a point, focusing on grant and financial management may improve performance; beyond that, it may interfere, instead. Numerous comments from SPs indicated that the total amount of compliance and accountability responsibilities required by both USAID and the 27 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGMPs interfered with their ability to carry out their work-plan activities in a timely fashion. In particular, SPs indicated that some UGMPs demanded a burdensome amount of supporting documentation for even small, routine expenditures; SPs felt it might have been possible to find some streamlined mechanism or procedure to simplify and expedite work. The starting point for GM and FM was for UGMPs to address findings from pre-assessments. Smaller, nascent NGOs typically presented more adverse findings from pre-assessments than larger, better established SPs. UGMP start-up workshops and trainings provided SPs with basic staff training and established guidelines and procedures to follow. Many SPs had to staff up and “skill up” FM and GM to meet USAID and UGMP requirements. For example, the RTC “RightMax” system required that SPs scan all supporting financial-report documents. Their SPs had to hire additional staff simply to keep abreast of the monthly scanning requirements, although once they hired additional staff and adjusted to Right Max, they began to see some benefits. Digitized financial systems appear to be the wave of the future, but SP experience suggests their introduction to SPs be handled “gently,” with more UGMP support during the introductory period. Compliance and accountability require that both UGMPs and SPs follow established process and perform their roles well. A key issue for SPs was the amount of time it took UGMPs to review and approve monthly financial reports and to disburse the next tranche of funds. A key issue from the perspective of the UGMPs was the extent to which SPs should compile and submit timely, accurate monthly financial reports with adequate supporting documentation. The “compounding” nature of the multiple sets of FM and GM requirements may have been a contributing factor in SP complaints that they had to produce enormous amounts of supporting documentation every month, and experienced extended waits for new disbursements. A series of SP staffing issues arose after the initial FM-GM training: • Many SPs reported they had to upgrade qualifications and pay off FM staff to meet the USG and UGMP requirements, either hiring new, higher level staff or retraining existing staff; • Some well-qualified staff found higher paying jobs and left; some FM staff committed fraud and had to be dismissed for cause • Replacing staff that left voluntarily or involuntarily was difficult and costly, especially for SPs that operate in smaller provincial towns or in rural areas. There is no “one-size-fits-all” approach to addressing SP FM and GM needs. UGMPs acknowledged that all SPs are different, and their GM-FM training needs are different. The UGMPs indicated that they made substantial efforts to tailor FM/GM training and support to each SP’s needs while complying with USG rules and regulations and with their own internal requirements. SP comments and documentation supported a conclusion that Pact was the most responsive to SPs and the most collaborative, followed by FHI360 and RTC. However, there was a range of responses about each UGM-partner. No UGMPs was rated as all good on every variable and none was rated as all bad on every variable. Conclusions UGMPs varied in the extent of autonomy from their own U.S. Headquarters (HQ) and in their ability to review and approve financial reports and grant other approvals quickly and efficiently. 28 South Africa Umbrella Grants Management (UGM) Performance Evaluation Local NGOs exercised total autonomy, in the sense that they were not linked to any U.S.-approval process; this is shown below. Figure1: Levels of UGM Financial Management Autonomy The South African UGMP, RTC, operated a fast approval process, in part due to digitization of supporting documents, rather than relying on paper receipts as the other two UGMPs did. Pact had greater autonomy and higher limits of authority from its U.S. HQ than FHI360, and was reasonably fast to approve. FHI360’s highly-centralized process required dual approval from both its South African and U.S. HQ, and was criticized by a number of SPs as slow to approve financial reports and disburse funds. SPs criticized RTC’s digitized, online, financial-management system, RightMax, as being burdensome, requiring investment in new scanning equipment and hiring of additional staff. However, once SPs adjusted to the demands of the system, it allowed them to track and to retrieve expense information more quickly and easily than before. FHI360 has tried to compensate for its dual review and approval requirement by using a daily “tracker” to monitor the status of review and approval of financial reports and disbursements. FM systems that rely on digitization of SP supporting documents make it easy for UGMPs to manage the review and retrieval of financial reports, but may impose new and unexpected demands on SP to add staff, increase skills, and procure equipment to comply with the system’s demands. UGMPs should alert and prepare SPs for the introduction of such systems. As attested by most SPs, improving FM-GM systems (including policies, procedures, and human resources) is key to becoming: (1) better positioned for external funding from other donors and (2) more able to expand services, coverage, and programming. Recommendations 1. Balance GM general objectives with FM compliance objectives USAID should ask future UGMP applicants to state how they would satisfy both USG and UGMP compliance requirements while remaining agile and responsive to SP GM-FM needs; for example, they could create an online, digitized, expenditure-documentation system that is easier for SPs to manage, but still provides faster filing and processing of supporting documentation. Future UGMPs should demonstrate the ability to manage and account for funds, but also work well with all SPs across a range of CB variables. Prospective UGMPs should demonstrate to USAID how quickly their systems can review and approve SP financial reports and supporting documents while ensuring adequate compliance. Local offices of international NGOs need to have adequate signatory authority to operate in a timely manner and their HQs should seek to streamline their involvement in RTC Fully autonomous (local) NGO – fast to approve (in principle) Pact SA Largely autonomous International NGO – Relatively fast approval FHI360 Highly centralized international NGO – slower to approve 29 South Africa Umbrella Grants Management (UGM) Performance Evaluation approval of financial reporting and expenditures, balancing compliance and accountability against timeliness. 2. Better "up front" SP understanding and commitment to sound FM and compliance. SP management should demonstrate that it is fully cognizant of and committed to the need for timely and accurate financial reporting and to providing adequate supporting documentation. The SPs present plans to acquire adequate staff, skills, and equipment (printer, scanner, etc.) to ensure timely reporting. Many SPs did not clearly understand or anticipate the demands they would face with USAID funding. USAID should give “fair warning” to potential SPs in the RFA to consider the need to increase staff and improve skill sets when they apply. UGMPs should reinforce this message. USAID should also make clear it expects well-qualified SPs that can handle all the demands associated with managing PEPFAR funding. For example, USAID should make a presentation at a pre-bid conference on its grant and financial management and regulatory requirements, and advise interested bidders to incorporate the costs of adhering to accountability and compliance requirements in their bids. The presentation should give examples of changes new USAID partners typically have to make. 3. Develop a more nuanced process for introducing new FM-GM information and reporting technology. Efforts to incorporate new FM-GM technology are essential to a culture of innovation and efficiency, as well as to being “green.” This would be in line with the call for increased use of technology by USAID Forward. An effort to balance the advantages of new technology with consideration of what SP can handle and how to introduce new systems and train partners is highly advisable. Suggestions above indicate there may be a need to assess SP readiness to adopt and implement new technology; this should include staff training in new skills to reach levels of computer or technology “literacy” to support all the changes. Partners reported that hiring and retaining staff to meet FM-GM demands of USAID and UGMPs is difficult. USAID should ask UGMP applicants to indicate how they might assist SPs with recruiting and retention, such as establishing or supporting a recruiting database that links qualified applicants with existing SP vacancies. Such databases may already exist in SA and could be readily adapted. Lessons Learned For USAID and other donors USAID Missions and other donors should consider carefully how to use a UGM mechanism to ensure accountability and compliance, as well as to build the capacity and enhance the performance of SPs. The first two objectives above are mandatory and cannot be reduced in any way; the second two objectives are to some extent elective, but equally important in the long term. Donors should be aware that UGMPs not only have to ensure accountability and compliance for the donor, but for themselves, in order to protect themselves against unallowable costs and the risk of 30 South Africa Umbrella Grants Management (UGM) Performance Evaluation large losses. This results in a situation in which the UGMPs have to impose on the SPs two sets of compliance and accountability requirements: external (donor) requirements and internal (their own) requirements. The combined burden of both sets of requirements falls on the SPs, who must satisfy both. Donors should strive for a UGM mechanism, and for UGMPs, that can meet compliance and accountability requirements, while balancing these against the need to be flexible and responsive, to set and use policies that encourage rather than stifle SPs, and to streamline procedures and turnaround times. Donors may wish to consider the lessons learned here, and to create online or digitized systems for sharing vital information more quickly, to support better decision making processes as well as better performance and management. Current information sharing seems to fall short of the optimal level. For Prospective or actual UGM partners or sub-partners NGOs considering applying for or now participating in a UGMP or SP role way want to consider the Lessons Learned, and take careful note of the difficulties and complexities associated with managing and reporting on the use of USAID resources. For SPs seeking USAID funding for the first time, it is important to note that USAID rules and regulations are demanding. They require a high level of effort and capacity for compliance and accountability. SPs should carefully consider the staffing requirements and systems needed to manage USAID resources. They should be prepared to make significant changes in organizational structure, policy, procedures, and systems in order to manage resources effectively, produce desired results, and report on achievements. CAPACITY BUILDING (CB) Capacity Building Provided by UGMPs to SPs and SSPs Findings In alignment with the emergency- and relief-funding model followed by PEPFAR, the bulk of UGM grants in South Africa have been aimed at scaling up the delivery of HIV and AIDS services nationwide. Therefore, consistent with the increased focus placed by PEPFAR Phase 2 and USAID Forward on enhancing sustainable local capacity—within both the national government and the civil society—the proportion of the UGM portfolio allocated to developmental-related activities and programs (e.g., CB2 ) has grown over time.3 2 APS 2006 defined CB as “activities that strengthen the skills of indigenous organizations to implement HIV and AIDS programs efficiently, with diminishing reliance on external technical assistance and support to a more sustainable HIV and AIDS response”. APS 2006 also stated that one of the two main UGM purposes (besides furthering scale-up of HIV and AIDS Services through local and international implementing partners) was “to develop indigenous capacity”. 3 Respondents across all UGM stress the fact that they have not received sufficient CB on governance and strategic planning. A few SP (Noah, South to South) stated that they did not receive any technical program capacity building. 31 South Africa Umbrella Grants Management (UGM) Performance Evaluation This especially has been the case for NGO CB provided in two of the seven areas identified in the UGM Annual Program Statement (APS) issued in 2006: 4 financial management and reporting of results. The ET's review of UGMP/SP reporting documents showed a decrease in non-compliance with UGM financial requirements, as well as the increase in timely submissions of quarterly progress reports among SPs between 2007 and 2012. This attested to the effectiveness of CB. However, the lack of adequate UGM indicators on CB outcomes (the APS included only one output-related indicator: the number of local organizations provided with technical assistance for institutional capacity building) did not allow a timely measurement of the progress—including strengths and weaknesses—achieved in other areas where respondents felt the capacity needs were the greatest, e.g., program management, strategic planning, and governance. Acknowledging the lack of clear processes in place to assess the effects of CB provided to grantees in the field, UGMPs (and SPs acting as umbrella organizations) developed a plethora of organizational-capacity-assessment tools. However, two major weaknesses were identified relating to the use of such tools: • Administration of tools was not as systematic and frequent as it should have been; UGMPs often did not provide thorough feedback to the SPs or SSPs on assessment results, especially on organizational self- assessments; • The information collected through these tools did not always inform development of tailored CB plans nor did it lead consistently to review of CB strategies not meeting the expected objectives. The most common modality to provide CB has been to organize workshops for multiple SPs. However, according to most UGM stakeholders interviewed, the most effective CB modality has been to combine customized CB support—e.g., mentoring and coaching—with multi-partner meetings where participants could share and learn from each other. Providing customized mentoring and coaching, however, has been constrained by locating the trainers and mentors in Pretoria, when the SPs are some distance away requiring travel and per diem expenses to "go on tour", and generally resulting in more set piece kinds of training workshops, rather than more personal problem solving and coaching. The CB provided by the four UGMPs to their SPs and SPP reflected different levels of expertise in a variety of areas. The workshops delivered by RTC and FHI 360 CAP demonstrated HIV/AIDS technical expertise in the areas in which they build capacity for, e.g., PTMTC, nutrition, treatment, care, and support. Both have focused more on building capacity in specialized technical skills rather than organizational capacity development. The CB provided by Pact seems to follow a clear and ambitious CB model and it relies on external (rather than in-house) technical assistance (TA) in many HIV/AIDS technical areas in which its SPs asked to be capacitated. The CB provided by 4 APS 2006 identified seven key CB areas: financial management, organizational management and governance, program management, quality assurance, strategic information and reporting, and leadership and coordination of partner organizations implementing prevention activities. 32 South Africa Umbrella Grants Management (UGM) Performance Evaluation FHI360 combined a relatively good level of CB experience and, according to the respondents, a moderate level of HIV/AIDS technical expertise. Besides providing direct (staff) CB support from UGMPs to SPs, the UGM mechanisms also provide “external” TA by contracting out for this expertise, as needed, to help their SPs. The advantages of external TA include reduced full-time staff cost, access to expertise on demand; and the ability to respond to almost any type of TA request. The UGMPs provide external TA in three different ways: 1. Pact: By recruiting and contracting external consultants and firms providing ad hoc support to SAG departments (DSD and DOH) ); 2. FHI360 and Pact: By recruiting and contracting external consultants in addition to the usual packages of CB support; and 3. FHI360 Capable Partners: By providing technical training and mentoring support to DOH specifically for the integration of nutrition and infant feeding guidelines into Prevention of Mother to Child Transmission (of HIV) (PMTCT) service delivery (see Appendix G for a detailed assessment of this program). All UGMPs provide some amount of external TA to SPs, especially in highly specialized areas, such as salary and remuneration. However, external TA, while helpful, was not consistent. Several SPs were unaware that external TA was available from their UGMPs; nearly all SPs were able to identify areas in which this TA would have been helpful. Other SPs asked for TA but did not receive it, although it had been agreed upon. The lack of clarity on why and when this additional TA is provided to SPs is puzzling, in light of reports of good communication overall between SPs and the Pact and FHI360 UGMP. In one instance, USAID reportedly arranged for Pact to fund a SP’s external technical support. However due to an oversight FHI360, the UGMP for this organization, was not informed of this arrangement. Table 1: External TA provided to SPs interviewed during evaluation FHI360 AAHT for strategic planning: AAHT chose the consultants Noah for salary equity review and RDQA: support provided by an HR consulting firm Pact CINDI and May’khethele Project: for M&E, Finance and program strengthening NACCW and Isibindi Project: across all 5 Pact CB areas During the first year of UGM implementation (2007–2008), UGM staff used to organize quarterly review meetings with SPs in the presence of USAID staff. However, such exchanges have not been frequent or regular over time. SPs strongly advocate for the revival of this practice. Other issues: • The criteria on what qualifies a UMG grantee to graduate from a SP to a prime are not clear to SPs and SSPs; • It appears SPs and SSPs that are most receptive to increased capacity development are the relatively young ones (less than five years of existence). Such organizations normally do not have fully developed systems in place. As they tend to have a relatively “empty toolbox,” (less sophisticated systems), they are therefore more eager to learn and adapt new systems. 33 South Africa Umbrella Grants Management (UGM) Performance Evaluation • UGM graduates want to receive limited CB support from their former UGMPs. • Pact views CB assessments as voluntary and partner-demand driven. RTC SPs indicated that some financial aspects of RTC CB were presented as compulsory, and that failure to comply with the assessment or follow up might lead to funding cuts or suspensions. Conclusions Capacity Building efforts for financial compliance and routine grant management have had a positive impact, in part because the UGMPs have a high stake in promoting good compliance, and in part because they are able to offer practical, standardized tools and templates for developing work plans, budgets, proposal writing and project planning. Other organizational development efforts related to strategic planning, governance, and other dimensions have not been as successful as expected both by the UGMPs and from the perspective of SPs. Set piece training programs on generic topics do not address the specific concerns of SP leaders. Selection of trainees by SPs and UGMPs is not well considered. Trainees on return to the work place may not have incentives to share their knowledge, or even to put in place new procedures or skills learned, and staff turnover means that training investments lack a long term payoff period. For small SPs, sending a key staff member to training means loss of time for the SP's work. There are many other reasons for this less than satisfactory conclusion, including: • Maintaining the core team of trainers/external technical assistance (TA) providers in Pretoria or Johannesburg, far from where most CB and mentoring needs occur, is not cost-effective or efficient, necessitating frequent travel and causing delays in SP training. It also contributes to the scheduling of training workshops at the convenience of the training officers, rather than the best time for busy and thinly staffed SPs. • Areas of CB needs tend to vary across SPs. UGM staff might not be able to cover all the PEPFAR technical domains under Prevention, Care, and Treatment as well as support all types of Capacity Building, as this covers quite a wide range of skills. • Exchange of lessons learned during implementation (a culture of “doing,” as opposed to a culture of “learning,”) needs to be promoted. Absence of learning options is a limiting factor and inhibits the building of a UGM “Community of Practice” (UGM COP). • It is necessary to find new ways to facilitate frequent workshop attendance (i.e., monthly) for smaller SPs with such limited staff that their absence stops work. Recommendations Strengthen UGMP's capacity to deliver customized CB trainings (supply side) that is both timely and responsive to the variable needs of SPs (demand side). A number of steps could be taken to do this: 1. Increase training and TA staff resources and utilization • Require each UGMP to hire or be readily able to source every type of TA. • Use the UGMPs in combination to tap each one’s technical expertise. USAID agreements would have to require each UGMP to help other UGMPs and SPs as needed. CB experts in OD areas should be available among UGM staff. A consultant database should also be compiled by or between UGMPs, and updated on a rolling basis to help SPs. 34 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2. Improve advance planning, scheduling and sequencing to maximize CB responsiveness and sensitivity to SP needs and workloads. • CB should include online, computer-assisted instruction (CAI), and should be as specific as possible to the needs of SPs and their staff. • UGMP should begin to develop CB plans right after the award of the UGM grant. A series of CB activities should be scheduled in advance and sequenced accordingly. • A systematic, online procedure to track and respond in a timely way to CB requests by SPs would be effective. Requests for CB should be well documented and feedback should be promptly provided. Scheduling CB further in advance and sequencing of CB activities and programs based on needs is key so that it does not interfere with field work. 3. Develop CB funding sources the use of which encourages SP demand side decision making, participation and ownership. • USAID should ask UGMPs to explore further the value FHI360 Education Training Fund for SPs, which offered financial support for training and workshop to enhance SP professional development. • Providing CB to an umbrella organization (HPCA, AAHT, CINDI, NACCW, and Noah) may be a more cost-effective way to reach more organizations than providing support to SPs one by one. 4. The UGM CB model should be based on the principles of adult learning (self-reflected, experiential, and participatory). It should be a “blended” model and encompass the three following activities: • Coaching: intensive, short-term CB focused on a specific area/topic, such as the financial and M&E requirements. Two coaching models (STEER Model or Solution focused OSKAR5) could be assessed and if found relevant an appropriate, they could be used. • Mentoring: Long-term CB focused on topics such as governance, resource mobilization, and strategic use of information. A mentoring plan would be developed for each organization. 5 See STEER and OSKAR for more details. 35 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Community learning events are collective, reflective moments for sharing experiences as and disseminating successful operational and M&E tools. These could take place quarterly in year one, semi-annually in the year two, and annually in later years. SPs that encounter challenges should be twinned with another SP that overcame that same issue. 5. Refocus CB and TA toward broader organizational development issues • The next UGM should to intensify CB in all seven key CB areas identified in the APS 2006 (financial management, organizational management and governance, program management, quality assurance, strategic information and reporting, and leadership and coordination). • UGMPs should provide “care for the caregiver” in the CB model for SPs serving OVC and PLWHA; good practices include Isibindi mentoring model. 6. Strengthen access to external TA for SPs • UGMPs should consider a transparent priority rating system to identify SPs most immediately in need of external TA. • UGMPs should link SPs to more intensive external TA and mentoring following training, especially for emerging/nascent SP. • UGMPs should consider a Board TA mentoring model to strengthen governance. • UGMPs should develop and manage a database of potential, qualified staff for SPs, as well as external TA consultants/companies, indicating key competency areas, (e.g., HR, salary structure/bands, job descriptions). • More external TA should be provided to DOH by FHI 360 Capable Partners (CAP). Good Practices The SI team was able to find examples of both UGMPs and SPs which took the initiative to develop good practices for improving their capacity to more effectively manage their organizations and to deliver services to their intended clients. Some of these "good practices" are highlighted below, and at the end of subsequent sections of this report. • Hospice and Palliative Care Association (HPCA) SSP progress in CB is rated on a five-star scale, based on standards for accreditation by the Council on Health Services Accreditation of South Africa (COHSASA) (The [HPCA] developed the standards in conjunction with COHSASA). • Kheth’Impilo has hired a skills development officer who conducted a skills audit for every employee (total of 1,028) and developed a database to track training for all staff. 36 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Noah developed a four-step assessment tool to gauge the level of organizational development across establishment, development, maturity and preparation for partnership, and independence. • Pact used a four-stage, organizational-development model: pioneering (nascent); rational (emerging); integrated (expanding); and associative (mature). • FHI360 used the Tuckman’s four stages of organizational or group development (forming, storming, norming, and performing). • The National Association of Child Care Workers (NACCW) developed a matrix to assess and rate the performance of Isibindi projects in a standardized manner. Lessons Learned For USAID and other donors Provision of CB and TA to SPs via UGMPs is a valuable adjunct to funding direct service delivery alone. For USAID missions, it reduces the management burden, while offering the potential to include more local partners in implementation. It also assures compliance with rules and regulations, as well as accountability for funds, and achievement of desired results. In addition, there is an opportunity to improve the performance of SPs. This opportunity can, in the best case scenario, include SP achievement of higher results, and better quality of program design and service delivery. However, in order for this potential to be realized, donors must plan carefully, requiring a baseline on clear indicators that capture and measure progress on desired variables, with repeat measurements at least annually. This level of measurement implies the need for a sophisticated and comprehensive M&E system and structure, with clear and meaningful reporting requirements. It also implies that donors can and will dedicate the time and effort to review and act upon data received to make informed decisions. Donors should also strive to calibrate the amount of funding set aside for CB and TA to the impact they desire to achieve in these areas, and should continue to fund CB and TA only to the extent to which they can measure and verify the impact of CB and TA against desired objectives. There are too many good, competing uses of donor funding to continue funding CB and TA at a set level simply due to precedent. For Prospective or actual UGM partners or sub-partners UGMPs and SPs, or those wishing to take on this role, should also study the findings, conclusions, and recommendations presented in this report. The promise of CB and TA is enticing; the implicit logic strongly implies that the act of simply providing or participating in these efforts will lead automatically to improved performance and great success in terms of fund-raising goals, achievement of results, quality of services, or long-term viability and sustainability. 37 South Africa Umbrella Grants Management (UGM) Performance Evaluation However, the evidence examined can neither confirm nor refute that this is the case: some SPs (the majority) grew and prospered, while others (the minority) encountered problems that, in the worst cases, reached critical stages. Statistically, then, it is clearly better to be offered and to benefit from CB and TA than not, but the evidence does not show it is either sufficient or necessary for success. At the same time, the absence of CB/TA does not lead inexorably to failure, but may reduce the chance of it. UGMPs should take note of the inherent difficulty of correlating the time, effort and expense of providing CB and TA with the results achieved and the impact created. Also, UGMPs and SPs should note the tremendous range of issues and concerns presented here, and consider the extent to which they might be likely to occur in other settings. UGMPs and SPs should be ready and able to recognize, address, and resolve such issues if necessary, as difficult as it might be to do so. DEFINING AND MEASURING SUCCESS (M&E) Findings APS 2006 stated that UGMPs would provide support to prevention, care, and treatment SPs in four key M&E areas: 1. Measurement of program progress: Little support was provided in this area, other than to measure the progress of CB activities; 2. Provision of feedback for accountability and quality: Feedback was provided to enhance compliance with existing financial and M&E requirements; 3. Surveillance: No specific support was provided in this area; and 4. Implementation of information management systems: With the exception of a few SPs, no specific support was provided for the development and implementation of IMS. Measurement of program progress Although SPs have been able to implement a variety of HIV/AIDS activities in different PEPFAR programmatic areas, SPs only were required to report progress made against a narrow list of PEPFAR indicators. As a result, SPs did not have many incentives to monitor and evaluate progress attained in other relevant and complementary areas. Similarly, the UGM M&E requirements, despite the revisions that occurred over time, do not seem to have adapted to the evolving features of the UGM mechanism, i.e., from being a grants management tool to a developmental mechanism committed to enhancing sustainable CB in a number of areas, including M&E, financial management, governance, and strategic planning. The area where the lack of adequate of indicators seems most apparent has been organizational development. Although SPs differ from each other in a number of variables (size, history, overall budget), no standardized tool was used systematically across UGM mechanisms to measure SPs and SSPs OD levels during the pre-award phase. In addition, no specific OD criteria were used to 38 South Africa Umbrella Grants Management (UGM) Performance Evaluation determine SPs eligibility for funding. Rather, areas of organizational weaknesses identified during the pre-award assessment phase were addressed at UGMP’s discretion. Furthermore, the OD measurement tools used during the pre-award assessment were not consistently employed for follow-up throughout the rest of UGM implementation period. The PEPFAR focus on quantitative and outputs-focused indicators was counterbalanced during the last year of implementation by two strategies adopted by SPs and SSPs to monitor the quality and effectiveness of their activities beyond the UGM requirements. Firstly, some SPs developed their own Monitoring, Evaluation, and Reporting (MER) plans, thanks to the support provided by their respective UGMP. However, in most cases, the MER are not being used, and data collected with the newly created tools is either not used for decision-making or only partially shared with the UGMPs in the narrative section their quarterly progress reports. Secondly, SPs and SSPs decided not to allow revision of PEPFAR indicator definitions to affect the quality of their service delivery. For example, SPs decided to continue to supply multiple services, even though the OVC service indicator changed from “number of OVC receiving three different types of services” to “number of OVC receiving at least one type of service.” As attested by UGMPs, the USAID AM, and the SPs themselves, (Khet’impilo, HPCA, NACCW, and CINDI - May-khetele), M&E capacity has increased over time, and compliance with UGM M&E reporting requirements has improved. That notwithstanding, data quality reviews (conducted occasionally by external entities) or the strategic use of collected information is not well integrated yet in SPs and SSPs day-to-day operations. Provision of feedback for accountability and quality To this end, interviewees suggested that SPs would have liked to receive timely feedback from UGMPs, not only on how to report their data, but also on how to use it for programmatic improvements. This need was felt, in particular, by SPs who hired M&E officers, but lacked prior formal training in M&E or data quality management experience. The relatively low level of M&E skills observed among some of the M&E officers may be linked to (1) low salary scales, and (2) the absence of Key Competency Areas (KCA) in the job descriptions (e.g. GRIP), against which staff performance could be assessed. Conclusions The recommended UGM indicators, i.e., the PEFPAR indicators, have been only moderately useful in measuring the overall mechanism’s effectiveness for two main reasons: first, the UGM M&E tools have been unable to adequately capture results attained in programmatic areas not directly associated with PEPFAR indicators. Second, despite revisions made to the M&E plans used by SPs over the last few years, the UGM M&E tools have persisted in measuring output-level results for service delivery, e.g., number of OVC served, or number of individuals trained. They have not monitored relevant SPs organizational-capacity development affecting service delivery, such as sustainability, communication, and governance. The main reason for this is the lack of qualitative indicators measuring outcome-level or process￾level variables, which would have allowed UGM stakeholders to appreciate UGMP effectiveness. The PEPFAR–driven M&E focus perpetuated the view of M&E as an accountability tool, rather 39 South Africa Umbrella Grants Management (UGM) Performance Evaluation than a vehicle of organizational learning, as recommended by the USAID Evaluation Policy (USAID 2011). There have been efforts to monitor the effects of CB conducted among SPs, as attested by tools developed by one UGMP (the Management Capacity Assessment Tool, the Organizational Capacity Assessment tool, the Program Capacity Assessment Tool, and the Monitoring, Evaluation, and Reporting Organizational Capacity Assessment Tools), Pact. However, such tools were not used consistently. Despite providing a baseline for SPs, they were not always followed up. And, despite their use to identify weaknesses, corrective actions, such as tailored coaching, was not consistently offered. Recommendations A comprehensive strategy for developing a practical and useful monitoring and evaluation system needs to be developed. This strategy, and subsequent implementation tactics should be the product of consultations with all stakeholders, and be designed to serve the information needs of decision makers at various levels, from USAID to the smallest SSP. An initial, step by step outline of the elements of such a strategy is presented below: • For USAID and other donors: It is critical for USAID and other donors to establish a set of clear indicators and insist on creation of M&E systems and structures to measure not only baselines and annual progress on CB, but also the extent to which CB contributes to improved performance, increased achievement of results, better service quality, and greater sustainability. All of these outcomes are implicit expectations that logically should occur as a result of in-depth CB across many factors, to justify the expense of CB. • Outcome indicators, then, should be incorporated into a comprehensive M&E and reporting system that is sophisticated and robust enough to collect and analyze the data to determine if the desired outcomes occur, and if they warrant the expense. Donors should require UGMPs to conduct an SP baseline that includes all indicators deemed necessary, as well as to measure and report progress at least annually. • Donors should also require regular and timely reporting on progress and issues. Here, “regular” means at least annually for routine indicators, and “timely” means essentially “Immediately” when UGMPs detect small problems and warning signs among their SPs that have the potential to spiral into big problems not detected and addressed quickly. Failure to act quickly can result in a crisis later. • Finally, donors should encourage a culture of evaluation and data use among UGMPs and SPs. This culture means creating a mentality in which data collection and its examination is considered essential for decision-making and management, and in which M&E is seen as an integral part of every activity. In the same way that no organization should spend even small sum of money without being able to account for it, UGMPs and SPs should strive to account for every activity in a meaningful way. • For Prospective or actual UGM partners or sub-partners: In a mirror image of the elevated place that donors should assign to M&E, UGMPs and SPs should also place a higher priority on M&E and reporting functions. It should be seen as an integral part of project design and implementation, nor as an add-on, afterthought, or a donor requirement. 40 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Large UGMPs and SPs should and indeed must have qualified staff that can design and manage an M&E system and reporting processes to meet management needs and donor requirements. Smaller UGMPs and SPs may need simplified M&E systems and structures which staff with limited qualifications can understand and manage, and which are proportionate in terms of the resources and level of effort. Investments in M&E, then, should be commensurate with the scope of the activity, the data needs, and the reporting requirements. • The ultimate aim of all UGMPs should be to support learning about, and management of, performance and progress, to help SPs identify and solve problems, and to make critical decisions in a timely manner. One challenge for UGMPs is to aggregate sufficient and meaningful data from their SPs to detect problems, manage activities, and document progress, without mandating the collection of voluminous amounts of interesting but essentially useless data. M&E Good Practices Some of the local M&E practices developed include: • A non-proprietary electronic patient record system (SOZO), developed by FPD for the public sector, has enrolled more than 110,000 patients to date; • Simplified M&E tools developed by Woz’obona to facilitate data collection by community care workers with limited literacy skills. • A user-friendly hospice data management system, developed by HPCA for M&E, was presented to USAID and subsequently rolled out to 58 hospices across the country. • DORA, a set of indicators developed by DOH for Regional Training Centers and DOH indicator dataset, which South to South uses. • A number of SPs and SSPs assess the performance of their staff against each of the key competency areas included in their respective Job Descriptions. CROSS-CUTTING THEMATIC AREAS Program Service Delivery Findings SPs vary widely with regard to whether, and to what extent, they received UGMP CB support to strengthen program service delivery. Overall, SPs who have received intensive program support (CINDI and NACCW) attest that it has strengthened all of their programs, not just PEPFAR￾funded activities. Apart from Pact, UGMPs have not appeared to emphasize CB support for program service delivery. Some SPs have been asked to move into new program areas outside their original mission in order to receive USAID PEPFAR funding, which has proven challenging for them; GRIP is an example of a grassroots organization that, despite 12 years of operation, still needs CB for basic 41 South Africa Umbrella Grants Management (UGM) Performance Evaluation organizational and program strengthening if it is to succeed in expanding into new areas of programmatic outreach (working with commercial sex workers in addition to rape survivors). Approximately one third of SPs interviewed stated that the relationship with their UGMPs had absolutely no influence on whether they strengthened the quality, scope, or reach of their program service delivery. SPs which provide program services outside PEPFAR indicators are at risk for being inadequately recognized for their program scope and good practices going undocumented. Measures of increase in number of programs or targets, or expansion of service delivery geographic focus, do not necessarily equate to the quality of sub partners’ service delivery; for some sub￾partners, expansion is not appropriate; for example, Woz’abona focuses on early childhood development (ECD) within a limited geographic scope. Conclusions Most of the CB effort has focused primarily on GM and FM practices necessary for compliance with USAID regulations, and secondly on more general organizational development dimensions. There has been far less effort to improve performance of service delivery by SPs. Even mature SPs need assistance with strengthening their ability to deliver HIV/AIDs clients with the variety of services that are now well tested and helpful. However, SPs have not managed to link their improved capacity for FM and GM into improved service delivery performance. Recommendations • UGMPs should emphasize quality program-service delivery as part of the pre-award assessment and CB model for SPs; this will be particularly valuable for nascent SPs and SSPs and reflects a growing national emphasis on sustainable programs and capacity development. • USAID should ensure that SPs funded through PEPFAR conduct a baseline study of scope and quality of services at the beginning of the funding period, as well as a follow-up evaluation at end of funding period. • UGMPs should establish twinning partnerships between SPs with demonstrated competence in quality program service delivery and innovative expansion into new program areas to mentor less experienced SPs. • Mentoring should also be built into the new CB model to assist SPs which need more support to strengthen program service delivery and should include ensuring that: the SP vision and mission informs program content, needs and feasibility, baseline data has been collected to inform program components and reach, services are being properly monitored and evaluated, and decisions to modify programs are evidence based. • UGMPs should encourage SPs to provide care for caregiver support in programs for staff working in front-line positions in HIV or OVC. 42 South Africa Umbrella Grants Management (UGM) Performance Evaluation • SPs working in schools or with seasonal workers should plan according to academic or harvesting calendars; this also applies to working with PDOH and DDOH to accommodate their annual planning commitments. • UGMPs should encourage SPs to attach summary documentation of non-PEPFAR-funded activities to quarterly PEPFAR reports to ensure understanding of the SP’s full range of program activities and implications for sustainability. • Based on the Agri-AIDS model, SPs working with farmers should consider mobile farmer￾to-farmer services, based on the GRIP model; other SPs working at the community level, or with migrating populations, e.g., commercial sex workers, may also benefit from mobile clinics. Good Practices Some of the higher functioning SPs, e.g., Kheth’Impilo and graduated partner HPCA, have been proactive in aligning their community-based services and strategies with the DOH PHC re￾engineering focus on expanding service at community level and strengthening linkages between health facilities and households. HPCA has been innovative in expanding its palliative care focus into prisons and schools. However, this programmatic expansion has not been influenced by the previous UGMP. HPCA pilots each new program and measures program results before rolling it out. For OVC programs, intergenerational services that also support granny caregivers or the entire household have been shown to benefit targeted children. A household focus also aligns with PHC re-engineering strategy to move more services to community level. Sustainability Findings SPs provided various definitions of sustainability: growth and continuity of the SPs, continuation of services regardless of the organization’s survival, and sustainability of initiatives and programs through transition from donor support to SA government ownership. The Isibindi Project is an example of the latter: the National Association of Child Care Workers (NACCW) is building a new, skilled, professional workforce, approved by national legislation, among unemployed community members. DSD at provincial levels have committed to funding the Isibindi project workers (child and youth care workers). Kheth’Impilo is also transitioning some of its community workers to DOH positions. PEPFAR funding is seen as having provided SPs with credibility to leverage other funding: “Knowing we met USAID standards was reassuring for other donors,” noted one SPs respondent. Sustainability usually is perceived in terms of financial sustainability. However, it also holds different connotations, including programmatic, institutional, and M&E as promoted by FHI360. As 43 South Africa Umbrella Grants Management (UGM) Performance Evaluation examples, HPCA is institutionalizing palliative care in health facilities, prisons, and schools and the Isibindi Project is training child and youth care workers who will work in government-funded programs; these activities were conceptualized as sustainability strategies. The most frequent sustainability-CB support provided to SPs by Pact and FHI360 has been fundraising workshops aimed at enhancing proposal-writing skills. Some, but not all, SPs have used these skills to develop proposals for new donors. A few SPs received external TA for institutional sustainability (NOAH received support to conduct an internal salary equity review in the hope that this would enhance staff retention and thereby influence organizational stability/sustainability). For a few SPs (CINDI and NACCW), CB for sustainability has targeted both SPs and SSPs; generally, this multi-tiered CB is perceived as having strengthened sustainability readiness for the SSPs. Most SPs interviewed, however, view development of strong relationships with government (DOH, DOE, DSD, DOA, Home Affairs, SASSA, Sports and Recreation) as the most effective sustainability strategy for their organizations, as this can lead to cost-sharing, direct funding, or strategic transition of staff from SPs to government payrolls. They emphasized “networking, advocacy, and fundraising vis-a-vis government” as the most important area for UGMP CB. This emphasis is borne out by the NACCW’s Isibindi Project, which has evolved over the past 10 years. Starting as an idea, the SPs created a model, demonstrated it, profiled it, got DSD to see its value; DSD has integrated it into Children’s Act legislation, declared it sustainable, and the Treasury has agreed to fund it. SPs requested specific sustainability TA beyond fundraising workshops: • OVC SPs request CB on understanding the DSD funding system, what funding they can request from the government, and government funding cycles, in addition to partnership￾building with government. • SPs which anticipate graduated status request CB focused on potential for organizational growth and resource diversification, and TA to position itself and access available funds. The Partnership Framework signed by SAG and USG in December 2010 indicated that a “wave of the future” will be to increase the focus on TA to government agencies that will take the lead in funding delivery of HIV/AIDS services. Conclusion Partners were very aware of the need to find long term funding sources, and for the most part believe that the SAG will be the best source of financial support over the long run. In that regard, SPs may benefit from a better understanding of SAG processes, and for better ways to access such funding programs should they exist. Recommendations • UGMPs should incorporate the SP requests noted above into the new, phased-CB model. 44 South Africa Umbrella Grants Management (UGM) Performance Evaluation • UGMPs should support donor-environment scans for SPs approaching graduation status, depending on organizational focus and services (for example, with corporate partners, municipalities in more prosperous provinces, and other bilateral donors). • UGMPs should develop measures for sustainability readiness as part of phased CB toward graduated status. • Future UGMPs should focus on supporting better strategic alliances between SPs and government to access funding opportunities. • UGMPs should expand upon public-private partnerships to leverage these resources. Good Practices • Two SPs employed “environmental scanning” to assess opportunities to sustain services through new partnerships, e.g., positioning to align with the new DOH PHC re-engineering strategies. • Noah capacitates SSPs (“Arks”) to function as independent and sustainable entities by moving them through a four-stage development process: (1) Establishment/Crawling; (2) Developing/Walking; (3) Maturing and Preparing for Partnership/Running; and (4) Independence/Winning. Independence is defined as the ability to adhere to sound administration policies and procedures, and secure a sustainable funding source. • Noah has also formed an endowment, the Noah Sustainability Trust, as a sustainability strategy. • The CINDI May’khethele Project has focused on strategies to ensure sustainability of school-based OVC services, including formation of HIV/AIDS committees comprised of principals, teachers, peer leaders, and ward councilors to strengthen schools’ capacity to continue program activities and the practice of linking schools directly with government departments to ensure the ongoing ability of students to access services. • Kheth’Impilo is exploring public/private sector opportunities and securing 18A tax-exempt status to make SPs more attractive to corporate, social-responsibility partners and has received pro bono, external, private-sector support for sustainability (provided by Bank of England). • Kheth’Impilo has also secured learning opportunities for its community workers to become phlebotomists or other health auxiliary workers; this builds career paths for rural community members and strengthens national human resources for health. • Graduated UGMPs, such as FPD and HPCSA, have developed self-funding, revenue￾generating arms. 45 South Africa Umbrella Grants Management (UGM) Performance Evaluation Communication and Coordination (C&C) Findings Given the triangular relationship between USAID, UGMPs, and SPs and the critical role of communication, the evaluation focused on related processes in this area. Overall, despite occasional problems, SPs indicated that C&C was satisfactory. Basic communication protocol dictated that USAID AORs communicate with UGMPs, and USAID AMs communicate with SPs, but keep UGMPs informed. Some UGMP protocols were overly ambitious, e.g. responding to SPs within 24 hours, and created expectations that could not always be met. Concerns regarding C&C included: • UGMP indicated an occasional concern with instances when SPs communicated directly with USAID or vice versa without informing them, leaving the UGMPs out of the loop. • UGMPs also indicated they would prefer to be consulted by USAID prior to decisions about annual SP funding levels, assignments, or reassignment, to assist UGMPs with planning. • RTC reported an instance in which USAID cut funds for Agri-Aids and informed Agri-Aids without looping in RTC. This created a brief crisis until partial funding later was restored. Other C&C issues were identified. Some larger SPs wanted to receive direct funding from USAID and did not appreciate being assigned to a UGMP. USAID expressed concern about its staffing capacity to manage such a large number of prime partners directly. This situation seemed to sour (slightly) SP perceptions of and relationships with the UGMPs. SPs also complained that UGMPs did not schedule CB training or visits with sufficient advance notice. SPs also noted that UGMPs initially provided little information about the CB process, including what might be available in terms of CB. SPs want to know at the onset what the UGMPs can offer them and to be consulted about scheduling. UGMPs indicated they provided CB based on assessments and needs, rather than offering a menu of options, to avoid creating unrealistic expectations or dependency. Some SPs indicated high satisfaction with the UGMPs. These SPs excelled at identifying and expressing their needs constructively, and pursued CB enthusiastically. Conclusion As might be expected, a triangular relationship requires a clear division of roles and responsibilities, and a very active and continuous communications process by which all receive relevant information in a timely manner. For the most part, the communication process worked well given the large number of actors, and activities that required daily attention. However, some negative events did occur where relevant stakeholders were not well informed. Even if few and far between, these events can 'sour' relationships and if not corrected, can lead to more serious problems. Recommendations • Communications between USAID, UGMPs, and SPs should follow established channels. Sidebars and jumping of levels, in the best interest of all concerned, is to be avoided. • USAID should consider ways to allow UGMP input into decisions on funding levels, while retaining the final decision authority so as to manage PEPFAR earmarks. 46 South Africa Umbrella Grants Management (UGM) Performance Evaluation • With SPs, UGMPs would be well advised to clarify from the start what is available, agree on priorities, agree what services SPs will be able to access, and manage expectations jointly. • SPs should emulate the approach of identifying needs clearly, expressing them to UGMPs, and advocating for the services they want. SPs should also make collective requests. • The next RFA should stress to potential UGMPs and SPs the importance of C&C to the success of CB, and to establishing and maintaining good relationships. • USAID should consider how to create rapid “360” feedback loops in future UGMPs, to provide concise, useful information about SP satisfaction and UGMP performance in “real￾time” (SP feedback about UGMPs shared with USAID and vice versa). • UGMPs should summarize and share online feedback from SPs about CB to identify and address concerns quickly, removing obstacles to performance. • In future work with DSD and DOH, USAID should seek to replicate Pact’s rapid C&C model (rapid development of TOR upon notice of SAG TA needs, followed by rapid posting of calls for bids, and ending with a rapid, inclusive selection process) and expand it to other SAG departments, focusing on annual planning, and ad hoc efforts, as needed. • UGMPs should communicate critical situations—looming crises—quickly to USAID to facilitate rapid and decisive interventions to limit risk, repair damage, and solve problems. Innovation Findings Based on case study interviews and documents reviewed, UGMPs and SPs employ numerous existing technologies. Uses of technology included: • Using scanners to digitize and bar-code invoices and other supporting documentation for financial reports • Designing software, such as the electronic, financial-management systems developed by RTC and Khet’impilo to track SP expenditures • Adapting M&E packages to the needs of HIV/AIDS sub-partners, such as Soweto-Care for Agri-Aids • A non-proprietary electronic patient record (SOZO) for the public sector developed by FPD • Electronic hospice data management system developed by HCPA In programming, Agri-Aids broke new ground by providing HIV prevention, counseling, and testing to farm workers. GRIP addressed the needs of adults and children who are rape survivors. Woz’abona provided quality Early Childhood Development and OVC services, NACCW pioneered a community-based child and youth care service delivery model, and USAID/Pact collaborated to contract expert, external technical assistance quickly and efficiently to assist DSD. 47 South Africa Umbrella Grants Management (UGM) Performance Evaluation Conclusions To obtain substantially different results, USAID could encourage future UGMPs to include new organizations, new approaches, and new technologies; for example, working more with the private sector to develop public-private partnerships, tap private sector resources, and access relevant private expertise to leverage new funds and ideas. With government taking the lead in funding direct-service delivery, NGOs must align their programs more carefully with government and access more government funding in order to reach their clients. Therefore, future UGMPs must develop the capacity of NGOs to align and work well with government. Recommendations USAID should encourage future UGMPs and partners to explore new technologies, programming approaches, business models, partnerships, and funding opportunities. Options might include: • Using online web applications to facilitate hiring, staffing, training, and retention of personnel • Exploring online tools, software, and communication technology to provide more mentoring • Developing new, secure, easier tools to review and approve monthly reports, and to disburse funds • Using online, pre-assessment tools to help USAID select a superior portfolio of SPs • Incorporating private-sector expertise into UGMPs to leverage resources and find new ideas UGM ASSISTANCE TO THE SAG: PACT AND CAP A Case Study: Pact’s External TA to SAG Pact’s UGM structure is unique, in that it includes a mechanism to recruit and fund external TA for SAG Departments of Social Development and Health, beyond the usual CB components found in the other UGMPs. Within the DSD, Pact has provided external TA for the Child Care and Protection and HIV and AIDS Directorates. Not all Directorates were aware that Pact is a UGMP. When a unit within DSD has identified a need for external technical support, it may inform USAID who will inform Pact; Pact will provide support to define the TOR if needed (reportedly, USAID has also provided this CB support to define needs at a national level), initiate a recruitment process, exert what appears to be variable influence on rating and selection of candidates in a joint process with DSD and USAID, track consultant deliverables, and initiate payment. DSD may assign a project manager who will also track the consultant’s progress. USAID support provided through Pact to DSD helps to fill budget gaps for: (1) external TA needs identified during development of annual operational plans, (2) identified TA needs not incorporated 48 South Africa Umbrella Grants Management (UGM) Performance Evaluation into operational plans due to their expected costs, and/or (3) ad hoc TA needs identified as programs evolve. This support is seen as quite useful for the department. Recent, external TA contributions through the UGM include (1) the development of an M&E framework for South Africa’s new Children’s Act that includes norms, standards and indicators and (2) development of a new costing tool for implementation of Children’s Act, the Child Abuse Neglect and Exploitation (CANE) strategy-costing tool for national and provincial DSD, and training in its use. This is the first time capacity for costing has been developed at the provincial level; once the tool has been approved by the Treasury, it can be used by other government offices as well as by DOH. External TA provided to DSD at national level is often intended to focus on, or cascade to, the provincial level. This is the level at which capacity needs to be built for implementation of new legislation, policies, trainings and guidelines. All three directorates identified a need for intensified capacity development (CD) at the provincial level. The external-support-funding mechanism through Pact is viewed as having contributed to improvements in quality of services, given that consultants play a major role in development of guidelines and standards intended to capacitate social service professionals to provide quality service delivery. Trainings provided by external technical experts for health professionals working at provincial, district, and sub district levels also serve to strengthen quality of programs and services. The HIV and AIDS Directorate has developed a CB and mentorship model to build capacity of the NGOs it partners; the model rests on four pillars: 1. Transfer of essential management skills through training 2. Mentorship in the application of the management skills at provincial level 3. Role clarification and mutually supportive coordination between Home and Community Based Care (HCBC) organizations and district-level government officials 4. M&E to monitor the progress and evaluate the outcomes of the project. The directorate hopes to discuss with Pact ways to align the capacity it provides to SPs with the unit’s plan to build capacity for partner NGOs. Conclusions: Perceived Strengths of Pact’s External TA to DSD Respect for SAG Overall, the USAID SA mission is appreciated for not attempting to “drive the agenda” for the DSD’s work when TA is provided, and is perceived as willing to engage in constructive dialogue to understand national plans and priorities and assess where they can be most helpful. Appreciation for the quality of relationships with both USAID and Pact colleagues was repeatedly emphasized. Flexibility and Rapid Response Reportedly, DSD policy does not allow use of the same consultant in consecutive years. This is perceived as unhelpful since, after consultants or firms have worked with DSD, their acquired expertise in departmental policies and systems, as well as the trust they establish at the provincial level, positions them as preferred experts. Pact’s ability to conduct an open bid and hire previous consultants, provided that they are the most qualified, is highly valued. 49 South Africa Umbrella Grants Management (UGM) Performance Evaluation While perceptions differ as to how much Pact influences consultant selection decisions, all directorates agree that Pact’s tender-response capacity is more rapid and flexible than the department’s internal TA processes (initiating TA takes three weeks to three months through Pact, as opposed to six months to two years through DSD internal processes, particularly for requests whose costs exceed R500,000). “Pact has a quicker way of doing things; government processes are long,” (DSD respondent). DSD reports Pact was also able to extend the scope of a TOR based on new needs that may emerge or components of work that may not have been anticipated initially. Clarity of Respective Roles The role clarity between USAID and Pact is appreciated (“USAID regards itself as the funder and Pact is the provider of technical support”), as well as the clarity between USAID/Pact and DSD: “They also understand their role and our role; they attend our meetings, understand DSD policies, programs,” (DSD respondents). Recommendations for Future External TA for DSD • USAID should encourage UGMPs and DSD to jointly develop a phased, comprehensive plan for provincial CB, . • Future UGMP TORs should require skills transfer from external TA to government counterparts so they acquire expertise, particularly at provincial level. • Future UGMPs should work more closely with DSD HIV and AIDS Prevention Unit to integrate proven components of UGM CB support into the unit’s CB model for NGOs. • Future UGMPs should consider any problematic hires over the past year when scoring and selecting consultants to better take into account DSD’s preferences and perspective. • UGMPs and USAID should participate with DSD HIV and AIDS Prevention Unit on technical/conceptual planning for national strategies, taking into account ‘social drivers.’ • Joint planning should include other key DSD stakeholders (e.g., UNICEF, UNAIDS, JICA). Strengths of the Current FHI360 CAP Training and Mentoring Model FHI360 CAP is well respected by national and provincial DOH as a provider of quality training aligned with international, national, and provincial policies and priorities. The organization is one of a handful of NGOs providing CB support to DOH in integrated PMTCT. While DOH values the contribution of FHI360 CAP, the department would not want an exclusive relationship with them as specialist TA provider; other organizations provide similar support. FHI360 CAP funding is not sufficient to meet national training needs. While FHI360 CAP activities at the central hub/facility level are limited, mainly due to capacity and budget constraints, at the sub-district level their impact has been significant in key areas, including improvements in exposed infant antibody testing at 18 months; increase in number of mothers who are exclusively breastfeeding; and an increase in early ANC bookings and in number of post-natal follow-up visits. 50 South Africa Umbrella Grants Management (UGM) Performance Evaluation At facility level, FHI360 CAP has focused on strengthening community clinics and involving community caregivers in behavior change and communication training to promote earlier ANC bookings, exclusive breastfeeding, and post natal care. In KZN, FHI360 CAP to CB of community caregivers was welcome. Conclusion: Challenges with the Current FHI360 CAP Training and Mentoring Model From the SI evaluation team’s perspective, one of FHI360 CAP’s main challenges is that it attempts to integrate national policies and guidelines from two key DOH directorates (PMTCT and Nutrition) as well as the HIV/AIDS Directorate. These entities do not always communicate effectively with one another, and have different objectives. The complex FHI360 CB model reflects an attempt to integrate policies from the various directorates into seamless program services, without the full collaboration from all relevant Directorates. FHI360 CAP has had limited success in strengthening the Regional Training Centers (Regional TC) through their training-of-trainer (TOT) program. Regional TCs lack sufficient capacity to take on a TOT role. Health workers in some provinces have been trained during the 10-day, integrated￾PMTCT training (core course), which was intended to capacitate them to train other health workers in the five-day, integrated-PMTCT training. However, DOH reported that in most instances, staff do not have time to train others, and in many cases do not have the necessary facilitation skills (this was not adequately covered in the 10 day training). Even if they were able to do the training, they would not have the time to do the on-site mentoring, an important feature of this model. Recommendations Strengthening the CAP Training and Mentoring Model will require: • USAID should consider retaining FHI360 CAP as an external TA service provider; • USAID should consider a more coordinated approach to integrated PMTCT CB with all relevant partners, for example by Two key activities are proposed: • Supporting an integrated annual PMTCT CB strategic planning process led by NDOH and including all specialist service providers in the field of integrated PMTCT CB; • Strengthen communication about FHI360 CAP and reporting to DOH at all levels. • Ensure that a final CAP report that fully documents the contributions is distributed within DOH at national, provincial, and district levels in all provinces where CAP operated; • Conduct an external final evaluation to provide evidence for wider replication of the FHI360 CAP training model by DOH; • Scale-up TA for the training of community health workers as part of the Primary Health Care Re-engineering strategy. support DOH in training of community caregivers; SUMMARY – LOOKING AHEAD Options for Future UGMs Based On Current Experience In response to USAID’s request for suggestions on how to implement UGM more effectively in the future, the SI team used the evidence and interviews conducted during the evaluation fieldwork to 51 South Africa Umbrella Grants Management (UGM) Performance Evaluation develop a menu of potential options for USAID to pursue in the future. USAID might choose to incorporate elements of any or all of the suggestions provided below. The SI team was encouraged to make recommendations, but was not privy to all necessary information to advise USAID on how to proceed with future procurement design, nor was the team composed of USAID procurement experts. With these necessary disclaimers, the SI evaluation team has sought to provide valid options; it will be up to USAID/SA to determine the merit and feasibility of the options presented here and make the final decisions. One UGM or Multiple UGMs? The advantages of multiple UGMPs include the power of competition, choice/alternatives among UGMs, and fail-safe options if a UGMP underperforms, or if a particular UGMP-SP combination proves to be unworkable. The advantages of only one UGMP, as suggested by the large UGMPs (Pact or FHI360), include a consolidated management structure, fewer management units for USAID, and, possibly, more centralized expertise under one roof—a “one-stop shop.” The disadvantage include the lack of alternatives if the UGM fails. However, a single prime could also consist of a consortium that leverages all the best features of the existing UGMs. Technical Expertise UGM or Organizational Development UGM? There are advantages of grouping SPs under a UGM by their PEPFAR-programming areas, for example, clinical partners (ART, PMTCT, and MMC) and non-clinical partners (Prevention, OVC, Home-Based Care, and Counseling and Testing). This would allow the UGM to increase staff in the relevant areas of expertise to provide TA to partners. However, not all partners fall neatly into clinical vs. non-clinical categories. Current experience suggests that even if they do, their CB needs may be very different. To group partners by OD levels and focus on CB needs would suggest a UGM strong in CB, but one that might not attend as well to partners’ needs for TA. This might result in their growing stronger as SPs, but not improving significantly in terms of quality of services, thus helping the SPs, but not the clients it serves. Geographic-based UGM or Funding-level-based UGM? Different UGMPs could handle distinct geographic areas. For example, USAID might suggest dividing the country into two or three zones (however, such a model would not be entirely compatible with the DOH priority districts). USAID could then solicit different UGMs to manage all the partners in each zone. This would address SP requests for more on-site TA and CB by allowing UGMPs to position more staff in the field, closer to the SPs and SSPs. However, not all partners fall neatly into particular geographic zones. How would USAID and UGMPs handle SPs and SSPs that operate in two or more zones? Alternatively, UGMPs could handle different SPs based on their respective size. One UGMP could deal with a few larger SPs and another one would handle small-to-medium-sized partners. The largest SPs would presumably be targeted to become primes, themselves, within a specified period of time by focus on diversifying resources and be weaned off reliance on USAID funding progressively as they develop other revenue streams. The small-medium SPs and SSPs would presumably need more CB to grow as organizations. In this case, the UGMPs would allocate more funding per SPs and SSPs and require more staff and/or consultant services to facilitate their development. 52 South Africa Umbrella Grants Management (UGM) Performance Evaluation Grouping Indigenous Partners with International Partners or Separating Them? USAID will seek more indigenous partners in the future, in line with its Partnership Framework and the USAID Forward plan. One question that arises is how will USAID manage those partners and develop their capacity? USAID staffing limitations require it to seek UGMPs to manage large numbers of smaller partners. USAID could group indigenous small partners with one or more large, South African SPs that have shown promise, as they might stronger affinity with each other than with an international SP. Examining the downside to this option, a South African SP might not yet possess all the experience required to manage USAID resources, and therefore might not be able to assist other local SPs in all areas. Rather, USAID could group indigenous SPs with an international UGMP that has more experience managing USAID resources for the time being. However, this approach implies that there are no local SPs that can manage USAID resources on a large scale, or help other local SPs to build capacity; the fact that FPD and HPCA have “graduated” to become direct partners suggests otherwise. Pre-defined or Open-ended UGM Application Process? USAID could outline some or all of the options above in an RFA, indicating the concerns it wishes applicants to address. USAID could indicate it would fund one or more UGMPs and invite applicants to propose models in reply. This might be attractive to USAID. It would put the burden on applicants to develop new models, form consortia, and use other strategies to address the concerns USAID faces. This might result in larger or indigenous organizations grouping together and forming partnerships that could also include smaller SPs, which are themselves umbrella organizations. These partnerships might address geographic dispersion of SPs and target population, TA needs, and large and small SPs. Applicants responding to an open-ended UGM application process might also propose to work with interesting groups or combinations, such as: • Rising Stars: One or more indigenous SPs on the verge of becoming “prime partners” requiring only limited support to stand on their own • Geographic or Affinity Clusters: New approaches to assist clusters of SPs based on geographical groupings, programmatic or technical needs, or “communities of practice” which learn together • Emerging Partners: New or small SPs which will require significant CB to grow and to develop, but whose programming or geographic location, i.e., remote areas, attract USAID • Local and International Groupings: Approaches that might pair local SPs with international SPs, or suggest a variety of approaches best-suited to the different needs of these SPs • Sustainability-Oriented Models, Government Alignment, and Public-Private Partnerships: This could include approaches designed to increase measurably the degree of SP sustainability on one or more dimensions—for example, by aligning them more closely with government strategies, or enabling them to develop public-private partnerships, possibly leading to “spin-off” organizations that could provide a marketable service or sell a 53 South Africa Umbrella Grants Management (UGM) Performance Evaluation product to raise revenue and turn a profit. The establishment of endowments or other long￾term funding options could also be considered. The SI evaluation team considers the last option, identifying the concerns that have to be addressed, then posing the challenge to applicants to propose the best model(s), to be the best way forward. Competition is a strong incentive to applicants to be creative and resourceful. USAID/SA does not have to answer all of the questions; rather, it has to identify all the questions, and pose them to the applicants to produce the answers. 54 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX A. SCOPE OF WORK UGM EVALUATION STATEMENT OF WORK 1. PURPOSE 1. The purpose of this external evaluation is as follows: Determine the extent to which the UGM mechanism has been progressing towards the achievement of its goals and objectives in grant making and capacity building, documenting what has worked and what has not worked and why; 2. Identify challenges in the areas of grant management and capacity building, and solutions applied in resolving them; 3. Analyze the UGM activities and recommend those that need to be modified or enhanced; 4. Assess the impact and benefits of investments on local organization capacity building; and, 5. Identify best practices and recommend strategies and priorities to inform future application of this mechanism as a development instrument. 6. The evaluation will review UGM activities that have been implemented to date, and identify successes, gaps and constraints in implementation. In addition, the evaluation will determine which results should be used to inform future UGM mechanisms. The evaluation team will be tasked with addressing the following questions: General • What key features of each UGM mechanism, enhanced or prohibited successful implementation and achievement of the key program objectives? • What key lessons can be drawn from the successes and or failures should be considered for future program planning? • Capacity Building • What key features of each UGM mechanism enhanced or prohibited capacity building efforts? • What were the key results, strengths and weaknesses and unanticipated results of the capacity development processes implemented by each UGM Mechanism under the grant management program? • What are the key lessons learned/promising practices that emerged from the UGM provision of capacity building support (e.g. in the areas of Monitoring and evaluation, finance, program management and implementation and organizational development)? • What indicators may be used to monitor a UGM’s capacity building work? Grants Management • What key elements in the internal management structure and systems of UGM mechanisms contributed (positively or negatively) to achievement or failure to achieve program results over the implementation period? • What key features of each UGM mechanism enhanced or prohibited grant management efforts? 55 South Africa Umbrella Grants Management (UGM) Performance Evaluation • What key elements and appropriate management structures have been most effective that may be incorporated into future grant management mechanisms? • What key lessons can be drawn from the successes and or failures that should be considered for future program planning? 2. ACTIVITIES This is a performance evaluation that will utilize the following data collection approaches: (1) review of background and source documentation; (2) implementation and analysis of an anonymous online survey to list of selected stakeholders; (3) “field observations” in which a predetermined set of key informant interviews are administered using a structured questionnaire; (4) delivery of a high-level evaluation briefing; and (5) finalization of draft evaluation report based on feedback documented during the briefing. The following is an outline of activities expected to be conducted by the evaluation team during the performance period: ACTIVITIES WEEK 1 WEEK 2 WEEK 3 WEEK 4 WEEK 5 WEEK 6 WEEK 7 Evaluation Team Arrives in Pretoria, South Africa On-Line Survey Review of documents Kick-off meetings, as required. The Evaluation Team will meet with USAID staff to review the scope of work and discuss the approach and objectives for interviews for major stakeholders will be finalized Team Planning Meetings and Work plan development, as required Meetings and Interviews with Key Stakeholders and Field Visits. This step will likely involve travel to places outside of Pretoria/Johannesburg which will include Durban, and Cape Town and one other site depending upon the final field observation plan agreed upon by the Evaluation Team. Data Analysis and Report writing Draft report submitted to USAID Submit final report and presentation to SA Mission 56 South Africa Umbrella Grants Management (UGM) Performance Evaluation 3. TEAM COMPOSITION The evaluation team will be composed of four consultants with significant knowledge about capacity building, project management in developing country settings and strong knowledge of HIV/AIDS and Orphans and Vulnerable Children program management and implementations. The team should have experience in the management of health and development programs in developing countries; excellent writing and communication skills with experience in producing team-based reports. The candidates must be able to work in a team to evaluate and synthesize information quickly, make clear and well-founded recommendations, and contribute to the written report and debriefings. The evaluation team should comprise of individuals with the following expertise: • Extensive development evaluation experience with substantial work experience in the South African and African context • Demonstrated experience in undertaking similar evaluations • Extensive capability building and grant management experience • Experience with HIV/AIDS and orphaned and vulnerable children’s programs and grant management expertise • Familiarity with the South African HIV/AIDS context, particularly the NGO sector • Extensive experience in employing both qualitative and quantitative data collection methods including participatory evaluation techniques 4. LOGISTICS A six-day work week is authorized if not in conflict with your organization’s policies regarding work week. Local holidays are not authorized. The evaluation team will be responsible for all off-shore and in-country logistical support. This includes international and in-country travel (including vehicle rentals), hotel bookings, working/office space, computers, printing and photocopying. The evaluation team, in collaboration with USAID/Southern Africa, will arrange all meetings, interviews, site visits, in-briefing and out-briefing. USAID/Southern African will provide all background documentation, online anonymous survey results, and contact information for suggested interviews. In all other respects, the evaluation team should be self￾sufficient. 5. REPORTS AND DELIVERABLES 1. Work Plan: After the Team Leader’s arrival in country a work plan will be developed during the team planning meetings and briefings with USAID/Southern Africa for approval by the Task Order COTR. 2. Debriefing: Prior to departure, the Evaluation Team will make a formal oral debriefing of approximately 45-60 minutes including a slide presentation, with an additional 60 minutes allowed for questions and comments at the USAID/Southern Africa Mission in Pretoria. Minutes of this meeting will be prepared by the contractor and attached as an Annex to the draft evaluation report. 57 South Africa Umbrella Grants Management (UGM) Performance Evaluation 3. Draft Evaluation Report: Prior to departure, the Team Leader will submit a draft evaluation report to the Task Order COTR that incorporates comments and information from the debriefing. The team will provide one hard copy and one electronic copy of the draft report. The report (not including annexes and attachments) will be no longer than 35 pages. It will include an Executive Summary, Introduction, Methodology, Findings, Lessons Learned, Conclusions and Prioritized Recommendation sections. The report is intended primarily for internal USAID use in assessing the performance of UGM mechanisms and defining future program needs. 4. Final Report: After the Evaluation Team submits the Draft Report, USAID/Southern Africa has 14 working days to review the Draft Report and provide a single set of written comments on the Draft Report. The Evaluation Team Leader will provide the Final Report no later than one week after USAID provides its comments. USAID/Southern Africa requests both an electronic version and five hard copies of the Final Report. 6. PROGRAM MANAGEMENT The USAID Technical Office responsible for the Task Order is the Program and Project Development Office (PPDO) at USAID/Southern Africa. The Contractor will report directly to the USAID Mission in Southern Africa through the COTR within PPDO. Reviewers will be provided with the following background documents in preparation for the assignment: • Original Annual Program Statement (APS) from USAID • All UGM original proposals • PowerPoint presentations to USAID used to award the UGM program • UGM MER Plans which has a set of program goal and objective indicator • Quarterly, Semi-Annual, and Annual Progress Reports • UGM implementation plan • UGM Annual work plans • UGM Country Operational Plan (COP) • Pact South Africa Evaluation Report March 2010 • AED UGM Mid-Term Evaluation May-July 2010 Partner documents: • Semi-annual and annual reports for UGM partners • Partner capacity building assessments and reassessments • Action plans for partners (based on the capacity building assessments) • Partner work plans • Partner Country Operational Plans (COPs) • Partner quarterly portfolio review 58 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX B. METHODOLOGY THE EVALUATION DESIGN The evaluation adopted a non-experimental case study design for four main reasons. First, an experimental or quasi-experimental design could have not been adequately captured the variety of organizational features (e.g., size, areas of intervention, years of activity, etc.) characterizing the SP supported by the UGM mechanism. By focusing on the distinctive features and uniqueness of sixteen different organizations (the so called case of interest) and, the design yielded valuable findings on a wide range of SP (see section 1 in each of the case studies included in the Appendix). The case study design is often regarded as a less rigorous design, due to the lack of a counterfactual evidence (the focus is on a small number of project beneficiaries rather than on their totality) and the relatively low generalizability of findings (the focus is on the specificity and uniqueness of the identified cases and not at all on how representative they are of the whole population affected by a program), , However, case study design can often be strengthened so as to enhance the validity and representativeness of its findings. In an effort to strengthen the validity of findings and conclusions, the SI team put in place two main strategies to mitigate what scientific literature usually refers to “validity threats”. The comparison across different UGM mechanisms (and therefore among several groups of SP receiving UGM grants according to different modalities) allowed the SI Team to control for a series of unobserved variables influencing the effectiveness of development interventions (e.g., history, maturation, and regression to the mean). The short timeframe, budget, and other constraints for this evaluation however did not make it possible to identify a control or comparison group (that is, to assess the performance of HIV/AIDS South African civil society organizations not benefiting from UGM grants), Comparison across UGMP facilitated the identification of distinctive and unique contributions p that USAID could take into account for design of the next UGM. Identification of a large number (16) of SP enhanced the generalizability of the evaluation findings. The SP sample represented a variety of geographical settings, target populations, and levels of organizational development attained. The use of triangulation (that is, the complementary use of secondary sources and an array of qualitative and quantitative data collection methods has also enhanced the validity of findings. Second, an experimental design (e.g., a randomized controlled trial) was not feasible as the UGM mechanism has already been implemented for several years without either baseline data collection or establishment of a control group. (i.e., the organizations receiving UGM funds has not been randomly assigned to the intervention, nor was there a comparison group of SP which did not receive funds). Third, a quasi-experimental design was not viable either as (i) the program has already been rolled out for several years and it would no longer be feasible to evaluate its effectiveness after determining eligibility to participate in based on any particular score (e.g., as in the case of regression discontinuity); and (ii) secondary data on HIV/AIDS organizations not benefiting from the project was not available for the sake of any statistical comparison (e.g., propensity score matching). 59 South Africa Umbrella Grants Management (UGM) Performance Evaluation Fourth, the nature of the questions included in the evaluation requested by USAID/South Africa did not focus so much on impact but on process-related issues and overall UGM effectiveness (e.g., the degree to which the program – across different UGM- enhanced SP grants management and service delivery capacity). This situation seemed to justify a non-experimental design. In the end, the case studies, as well as the online survey, and other documentation constituted the backbone of the overall evaluation. By comparing the findings of the case studies against a common set of topics identified with the USAID/South Africa (and hereby referred to as “emerging themes”), the 16 case studies helped to guide the development of the different report sections (findings, conclusions and recommendations) for each of the identified themes. SAMPLING STRATEGY The online survey developed by the SI Team before in-country data collection was administered to all SP, with a response rate of 75%. Therefore, as the whole universe or SP population of interest received the survey, no specific sampling strategy was used. However, USAID managed the on-line survey, and the results were not included into the final report. The same applies to the semi-structured interviews conducted with USAID staff involved in the UGM Program (the UGM Agreement Officer’s Technical Representative (AOTR) as well as all USAID Activity Managers (AMs), including PEPFAR Liaison Coordinator). For the sake of the case study development, 16 different organizations were identified based on the combination of an extreme case and critical case sampling strategies. Put simply, case studies were developed on those SP which proved to perform the best and the least and seemed to be the most relevant for programmatic purposes. The rationale for exploring in details the experience of the “best cases” was to identify best and worst practices and organizational features that might contribute to success or failure, so as to promote/discourage their replication among SP in the future. The rationale for documenting the experience of the least performing SP was to identify organizational dynamics and contextual factors hindering the success or contributing to the failure of UGM grantees. This would allow compiling of a list of “early warning signs” as well as suggest a few prevention or mitigation strategies for USAID to consider in the future. In the course of case study development, attention was paid to innovations (both programmatic and technological) introduced by the UGMP and SP included in the sample. The validity of such findings was strengthened by using interviews (i.e., recall), or historical program documents (i.e., annual reports) to measure or compare at multiple points in time the performance of each SP throughout the period of UGM implementation. For each SP identified, the evaluation team met with its management/coordinating team. This would include the SP COP or Executive Director and key staff, as well as the UGM COP or Director, the Financial Officer and the M&E Officer. With respect to the interviews with the government officers, both at the national, provincial and district level, a purposeful sampling strategy was adopted: national DSD and DoH officials who collaborated with the UGM (e.g., officials who benefited from external technical assistance or attended a training provided by one of the UGMP) were interviewed in Pretoria. Similarly, provincial and district government officials (both at the DoH and DSD level) were interviewed in the Western Cape and Kwa Zulu Natal). 60 South Africa Umbrella Grants Management (UGM) Performance Evaluation DATA ANALYSIS Due to the nature of the case study design, the evaluation mainly relied on qualitative data analysis. Based on an in-depth review of UGM-related documentation as well as a preliminary analysis of findings from both the online survey administered to all SP before the start of data collection and the open-ended interviews held with UGMP during the first week in country (mixed methods), a number of key UGM topics (“emerging themes”) was identified. Sixteen SP were selected for case studies that would allow the SI Team to learn how each UGM mechanism fared on each of the identified topics (qualitative data analysis step 1: identification). In developing each case study, each evaluation team used the same template. Although each draft was 10 pages long on average, the evaluation team opted for a much shorter version (2.5 pages) to include in the final report (qualitative analysis step 3: reducing). For each case study, the team referred to two different documents: (i) The complete/detailed list of notes; and (ii) the condensed version of the notes based on the template (2.5-3 pages (qualitative data analysis step 1: sorting). Exploring the identified emerging themes through SP staff’s narratives and recounts of their personal experiences allowed highlighting some complementary themes, thus generating further learning, along the process (what literature refers to as the “constant comparison method”). In order to enhance a participatory data analysis and stimulate group analysis, team members, each working on four case studies): (i) shared their preliminary conclusions with the group; and (ii) signaled any new theme emerging on top of the eight which had already been identified. By providing a snapshot of the experience of each SP in relation to the identified emerging themes (qualitative data analysis step 2: organizing), the mini-case studies allowed the team to generalize some of the findings. This was also facilitated by a four-day intensive session of group data analysis, held in Pretoria upon return from the field. The SI team also triangulated findings using multiple sources (program documents, semi-structured interviews with several stakeholders at multiple levels, key informant interviews both within and outside of USAID). OTHER EVALUATION METHODOLOGY FEATURES ENHANCING THE RIGOR OF FINDINGS Limitation of Bias The use of triangulation (that is, the presentation of findings derived from the convergence of multiple methods, both quantitative and qualitative) has contributed to limiting the researcher’s bias. Likewise, utilization of a common template and clearly articulated guidelines towards the case studies development enhanced the comparability of SP experiences, thus facilitating the finalization of the report sections on the emerging themes. 61 South Africa Umbrella Grants Management (UGM) Performance Evaluation Subjective judgments The SI Team established a “member check-in” mechanism within the team (regular, in-depth calls when the team was apart or face-to-face meetings when the team was together) and had regular weekly meetings with the client, USAID. Each evaluation team member as well as USAID/South Africa had the opportunity to review the preliminary findings and to ask that those not clearly backed up by evidence be revised). Evaluation team members followed up with the evaluation stakeholders (e.g., UGMP and SP meet during semi-structured interviews or the field visits to their project sites) as needed for clarification or validation. In drafting the findings, the evaluation team also made a specific effort to maximize use of documents (e.g., training curricula, training modules, annual reports) that provided relevant information. Conflict of interest Overall, the SI team did not have any specific conflict of interest vis-à-vis the UGM program being evaluated. However, as one of the evaluation team members (the local South African OVC specialist) had carried out a three-week consulting assignment for one of the UGMP (Pact), Social Impact made sure that she not be involved either in the data collection or data analysis related to the UGMP question, as attested by the specific clause included in her contract. Consistent with the Social Impact Ethics Code and USAID Evaluation Policy, the consultant was asked to leave the room in the course of meetings or interviews any time any issue related to the UGMP in question was discussed. 62 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX C: SI EVALUATION REPORT STANDARDS OF EVIDENCE To ensure the validity of findings, conclusions, and recommendations, SI maintains the following standards of evidence for its evaluation reports; adherence to these standards is cross-checked by SI HQ staff who review evaluation team drafts submitted after fieldwork is completed, with later revision made as necessary: • Findings: Findings must be specific and concise, and supported by quantitative and qualitative evidence. In this case, quantitative information was drawn from USAID, UGMP, and SP project documentation; qualitative information was drawn from case studies of UGMPs and SPs, and key informant interviews with all USAID AOTRs and AMs associated with the UGM mechanism. • The report contains confirmatory evidence from multiple sources, data collection methods, and analytic procedures. In this report, the multiple sources include: USAID, UGMP, and SP documents and data; an online survey of SPs; case study interviews; as well as SI evaluation team data analysis • Analysis: Analysis conducted by the SI evaluation team included the following procedures: • Conclusions: Conclusions are drawn logically from findings and analysis. In this evaluation report, the SI evaluation team made every effort to draw conclusions as logically and directly as possible from the evidence gathered and data analyzed. • Recommendations: Recommendations are supported by specific findings and conclusions, are practical and specific, responsive to the purpose of the evaluation, action oriented, and indicate the person or entity that is responsible for each action. In this report, the SI evaluation team strove to meet this test for all recommendations, indicating whether recommended actions should be taken by USAID, the UGMPs, or the SPs, or some combination thereof. • Lessons Learned: The report provides lessons learned that will help not only the immediate client, USAID/SA, but also USAID and other donors to determine how, when, and where to use a UGM to manage a large program portfolio, an enormous budget, and large numbers of local partners. It also provides information that may be useful not only to the UGMPs and SPs involved in South Africa, but in other countries where a donor may decide to utilize a similar UGM mechanism. 63 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX D. PERSONS CONTACTED MEETINGS, INTERVIEWS, SITE VISITS DATE USAID BRIEFINGS In-Brief: Mini-In-brief – Program Office, Charles Mandivenyi; Anita Sampson, other Health Office staff Mission In-brief - MD Jeff Borns, Dep. MD Cathy Moore, Program Office & Health Office Staff 12 March 12 March Cephas Goldman – AM 14 March Wendy Benzerga - Global Fellow 14 March Malik Jaffer – AM/AOTR 14 March Anita Sampson, AOTR 14 March John Kuehnle – AM 14 March Thobikile Finger - AM 14 March Olga Mashia, USAID –AM 15 March Thapelo Maotoe – AM 15 March Lauren Marks – AM 15 March Anita Sampson/Charles Mandivenyi 16 March Christian Chappelle – Health Officer 20 March Out-Brief and Presentation: MD Jeff Borns, Dep. MD Cathy Moore, Program & Health Office Staff 11 April USAID WEEKLY CHECK-INS Anita Sampson/Charles Mandivenyi 15 March Anita Sampson/Charles Mandivenyi 23 March Anita Sampson/Charles Mandivenyi 30 March Anita Sampson (Charles Mandivenyi by email copy) 3 April Anita Sampson (Charles Mandivenyi by email copy) 10 April 64 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM MEETINGS 3.1 Introductory UGM meetings FHI 360 (COP Ruth Mufute and key staff) 12 March FHI 360 Capable Partners (COP Phyllis Baxen and key staff) 12 March Pact (COP Malika Magagula and key staff) 13 March RTC (COO, Kurt Firnhaber, Program Manager, Gert Van de Merwe, other key staff) 13 March 3.2 Follow-up UGM meetings FHI 360 - Sent follow up questionnaire By email 26 March FHI Capable Partners - Sent follow up questionnaire By email 26 March Pact Daniel Bakken & Mathabo Mathabo re Contracts COP Malika Magagula & staff re CWSA Sent follow up questionnaire 19 March 19 March 26 March RTC Sent follow up questionnaire By email 26 March GOVERNMENT MEETINGS – DOH/DSD CAP Stakeholders KZN District DOH, Dudu Ntombela, Manager, PMTCT Umkhanyakude District 19 March National DOH, Precious Robinson,, Deputy Director PMTCT 20 March National DOH – Nutrition Ms. Lynn Moeng, Director; Ms. Ann Behr, Dep. Director 20 March KZN Provincial DOH, Lenore Spies, Manager, Nutrition 22 March Western Cape DOH: Mr. Stephen Titus, Director, Comprehensive Health; Mrs. Hillary Goeiman, Dep. Director, Nutrition; Mrs. Nicolette Henney, Assistant Director, Nutrition; Mrs. Dyes Hana Manjekane, Dep. Director, HIV Prevention 2 April 3 April Pact Stakeholders National DSD (HIV Prevention Directorate Chief, Ms. Connie Kganakga and members of staff) 23 March 65 South Africa Umbrella Grants Management (UGM) Performance Evaluation SUB-PARTNER MEETINGS/SITE VISITS Pact Sub-partners CINDI 19 March Keth’impilo 26 or 27 March NACCW 30 March Childline Mpumalanga 29 March World Vision US 3 April FHI 360 Sub-partners Noah 22 March Anglican AIDS 2 April Hospice Palliative Care Assoc. 2 April Humana 27 March Grip 30 March Woz’obona 26 March – Pretoria 2 April - Limpopo Right to Care Sub-partners Agri-Aids on 20 March 20 March - Pretoria 29 March – Nelspruit area South to South 27 March External technical service providers Theresa Wilson 23 March Tim Wilson 3 April 66 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX E. SURVEYS/QUESTIONNAIRES 1. ONLINE SURVEY 2. UGM QUESTIONNAIRE 3. SUB-PARTNER QUESTIONNAIRE a) FHI360 CAPABLE PARTNER QUESTIONNAIRE b) NDOH QUESTIONNAIRE c) PDOH QUESTIONNAIRE d) FOLLOW UP QUESTIONNAIRE e) CAP ONLINE SURVEY 4. FHI360 CAP SITE VISIT QUESTIONNAIRE 5. DSD QUESTIONNAIRE 6. UGM FOLLOW-UP QUESTIONNAIRE 7. GRADUATED SUB-PARTNERS 67 South Africa Umbrella Grants Management (UGM) Performance Evaluation I. ONLINE SURVEY UGM EVALUATION Suggested List of Questions for Online Survey submitted by Social Impact Evaluation Team to USAID/Southern Africa (March 2, 2012) 1. What were your organization’s objectives when you first applied for a UGM grant? Please rank your responses in order of importance: First Objective………………………………………………………. Second Objective…………………………………………………. Third Objective …………………………………………………... Fourth Objective ………………………………………………..…. Fifth Objective ………………………………………………………. 2. Based on your organization’s experience with the UGM grant over the last few years, to what extent were your initial objectives fulfilled? VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE First o o o o o Second o o o o o Third o o o o o Fourth o o o o o Fifth o o o o o Please describe what exceeded your expectations the most:………………………………….. Please describe what exceeded your expectations the least:………………………………….. 3. a) To what extent did the UGM grant enable your organisation to improve the quality of its service delivery? VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:………………………………………………………………………………………… 68 South Africa Umbrella Grants Management (UGM) Performance Evaluation b) How likely is it that you will be able to sustain the current quality of your service delivery after the UGM grant ends? VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O 4. What is the contribution of the UGM grant to your overall budget/funding needs? O Less than 10% O Between 10% and 25% O Between 26% and 50% O Between 51% and 75% O Between 76% and 100% 5. To what extent did the UGM grant improve your organization’s ability to diversify your funding sources? VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:………………………………………………………………………………………… 6. How did the UGM grant affect the coverage of your organization’s activities in the following four areas? a) Increased Number of clients VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:………………………………………………………………………………………… 69 South Africa Umbrella Grants Management (UGM) Performance Evaluation b) Increased Size of service area VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:………………………………………………………………………………………… c) Increased Number of services VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:………………………………………………………………………………………… d) Increased Type of service VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O Please describe:…………………………………………………………………………………………… 7. How likely is that you will sustain increases in coverage of your organization’s activities after the UGM grant? a) Number of clients VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O b) Size of service area VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O c) Number of services VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O 70 South Africa Umbrella Grants Management (UGM) Performance Evaluation d) Type of service VERY MUCH MUCH UNCERTAIN LITTLE VERY LITTLE I DO NOT KNOW O O O O O O 8. USAID is interested in building upon those aspects of the UGM mechanism that grantees have found the most beneficial. Please describe up to three aspects that your organization found particularly useful (if any): 1. ………………………………………………… 2. ………………………………………………… 3.………………………………………………… 9. USAID is interested in addressing those aspects of the UGM mechanism that grantees have found the least beneficial. Please describe up to three aspect(s) that your organization has found particularly challenging (if any): 1. ………………………………………………… 2. ………………………………………………… 3.………………………………………………….. 10. UGM mechanisms were set up to facilitate the timely channeling of funds to enhance an effective national response to HIV and AIDS. To what extent do you believe that UGM was timely in the following three areas: a) Length of the Grant application review process? VERY TIMELY TIMELY UNCERTAIN NOT VERY TIMELY NOT AT ALL TIMELY I DO NOT KNOW O O O O O O b) Grant Disbursement after award? VERY TIMELY TIMELY UNCERTAIN NOT VERY TIMELY NOT AT ALL TIMELY I DO NOT KNOW O O O O O O 71 South Africa Umbrella Grants Management (UGM) Performance Evaluation c) UGM Staff responses to your queries? VERY TIMELY TIMELY UNCERTAIN NOT VERY TIMELY NOT AT ALL TIMELY I DO NOT KNOW O O O O O O 11. Were there any unanticipated results that occurred as a result of the capacity development processes implemented through UGM grant management? If so, please describe. 12. What recommendations do you have, if any, for USAID to strengthen or make UGM funding mechanisms more successful in the future? THANK YOU VERY MUCH FOR COMPLETING THE SURVEY. IF YOU WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE WRITE TO THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM ALL YOUR E-MAIL EXCHANGES WITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NEVER BE MENTIONED IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 72 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM QUESTIONNAIRE USAID South Africa UGM Evaluation – Key Informant Questions – UGM Partners The evaluation team for the USAID South Africa UGM mechanism seeks to learn about the partnership between UGM Partners and USAID, and between UGM partners and their subgrantees. This evaluation focuses on Free State, North West, Limpopo and Western Cape. We would like to ask you some questions and appreciate your taking the time to respond, to help us to define next steps as we begin to speak with UGM managers and staff, government counterparts and other stakeholders, and health care workers. Option A: USAID South Africa is with us today. We can ask these questions, if we feel they are comfortable answering. We also invite USAID SA to comment if they feel it is appropriate. Otherwise we can shift to appreciative inquiry if that seem a better way. Option B: USAID South Africa is not with us today. We will ask you the questions, and we encourage you to answer as frankly as you can. We will not share answers with USAID. First Section General questions to be discussed with the UGM Chief of Party or Project Director and key staff (Identify each respondent – Name, Title, email address, and phone number on sign-up list). 1. (Beginning with the Chief of Party or Project Director and then taking other key staff in turns) Can you explain to us briefly, first, when and how your organization obtained a UGM award, and second, how your organization went about the process of selecting sub-grantees? Follow up questions: i. When they first applied for a UGM award/when they renewed; ii. Describe the process set up by USAID, and if possible, tell us how long it took from time of proposal to time of award. iii. Also let us know, if possible, about how long from award notification to receipt of funding in the first year. iv. Is the funding cycle regular each year? Or does it depend on PEPFAR funding cycles? How do you manage if there are delays in funding? 2. Your funding is under an APS. This must have required certain initial approvals from USAID as well as annual approvals. Can you describe the approvals that were/are required, and the process for obtaining them? How long did it take to obtain approvals? (This can be stated in terms of the time and/or effort required to obtain each of the approvals) Follow up questions about approvals (address each of the following if possible): i. Key Personnel _______ ii. Work plan(s) _______ iii. Proposed grant criteria _______ iv. Proposed grantees ___________ 73 South Africa Umbrella Grants Management (UGM) Performance Evaluation v. Other PEPFAR or USAID requirements __________________________ • Semi-annual Reporting; Other reporting • Annual COP submissions • Incremental funding requests? • Accruals – if any? • Other requirements? 3. Project start up is an inherently time-consuming process. We are interested in how long start-up took for your organization, and how long it took for your subgrantees. 4. In addition to time needed to obtain the initial approvals above, what other start-up tasks were required (i.e., office rental (include main office and any sub-offices), vehicle and equipment procurement, recruiting, hiring and training staff, and so on). 5. Can you tell us about how long overall it took for these start up tasks in order for your UGM to be fully operational? 6. Since the UGM became fully operational, have you remained at full force since then, or has the project experienced any other major delays or setbacks? If so, can you tell us what they were? How did you respond to them? 7. Can you tell us on average how long it has taken for subgrant-funded activities to become fully operational? For those which took longer than average, why was that? 8. Once the subgrants became fully operational, in general, have they remained at full force, or have there been delays or setbacks with some partners? Can you explain? 9. How closely would you say the UGM project has followed the UGM objectives as proposed, and how much have the goals and objectives changed over time, if at all? VERY CLOSELY CLOSELY SOMEWHAT NOT VERY CLOSELY NOT AT ALL CLOSELY MAJOR CHANGES OCCURRED O O O O O O 10. If goals and objectives occurred over time, can you describe what changed, when, and why? Were the changes initiated by you, or requested by USAID, or the grantee(s)? 11. Have the expected results remained the same or changed since the beginning? TOTALLY CHANGED MAJOR CHANGES A GOOD BIT CHANGED SLIGHTLY UNCHANGED O O O O O 12. For any important changes, what challenges did the changes present for successful implementation of the UGM? 74 South Africa Umbrella Grants Management (UGM) Performance Evaluation 13. Regarding the objectives and desired results, which of them have proven to be the easiest to achieve so far, and which have been particularly challenging? 14. Can you describe what criteria and steps you envisioned in the beginning to assess and select sub-grantees, and can you describe how those criteria and steps have evolved? If you changed criteria or steps, can you tell us why? 15. How clear were the USAID funds management and reporting expectations between your organization and USAID, and between your organization and sub-grantees since the beginning of the program? (What issues if any arose?) Level of clarity with (Tick the best response in each column): USAID GRANTEES Very Clear Very Clear Clear Clear Neutral Neutral Unclear Unclear Very unclear Very unclear Do not know Do not know What have you found most challenging? How did you handle the challenges? 16. From the start, how clear was the understanding between your organization and USAID about the Capacity building activities to be undertaken, and in general, how clear was it between your organization and UGM sub-grantees? Please explain. Level of clarity with (Tick the best response in each column): USAID GRANTEES Very Clear Very Clear Clear Clear Neutral Neutral Unclear Unclear Very unclear Very unclear Do not know Do not know 75 South Africa Umbrella Grants Management (UGM) Performance Evaluation 17. From the start, how clear was the anticipated number or volume of UGM sub-grantees or sub-grants for which you would provide grant management and capacity building support? (If expectations changed over time, please say when and why) VERY CLEAR CLEAR NEUTRAL UNCLEAR VERY UNCLEAR DO NOT KNOW O O O O O O 18. From the start, how clear were USAID/PEPFAR Monitoring and evaluation requirements for your organization, and for the subgrantee management of the UGM? How, when and why did they change? Level of clarity with (Tick the best response in each column): USAID GRANTEES Very Clear Very Clear Clear Clear Neutral Neutral Unclear Unclear Very unclear Very unclear Do not know Do not know Communication and coordination with USAID and sub-grantees would be important elements of project success. You have previously indicated the types and frequency of communication you used with sub-grantees. Overall how effectively do you feel your communications with sub-grantees have been – both those initiated by you, and those initiated by your sub-grantees. What would you change, and why? Responses could include technological barriers, limited time, etc. VERY EFFECTIVE EFFECTIVE MIXED RESULTS SOMEWHAT INEFFECTIVE INEFFECTIVE O O O O O Overall, how effective do you feel that communication and coordination with USAID have been – both those efforts initiated by you, and those initiated by USAID? What would you change, and why? 76 South Africa Umbrella Grants Management (UGM) Performance Evaluation THANK YOU VERY MUCH FOR YOUR TIME. IF YOU HAVE ANY QUESTIONS ABOUT THE EVALUATION PROCESS, OR WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE FEEL FREE TO COMMENT NOW, OR WRITE TO US AT THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM ALL YOUR COMMENTS AND E-MAILSWITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT APPEAR IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 77 South Africa Umbrella Grants Management (UGM) Performance Evaluation SUB-PARTNER QUESTIONNAIRE USAID/South Africa UGM Evaluation Semi-Structured Interview for Sub-partners Organization:_________________________________________________ Location: ____________________________________________________ Umbrella Partner ☐Pact ☐FHI ☐RTC RESPONDENT Director Finance Officer M&E Officer Technical Other Names: _____________________________________________________ Date/Time and Duration: ________________________________________ Section 1: Introduction Good morning, My name is__________________________ and I am a member of the Social Impact Evaluation Team currently assessing the performance of the HIV/AIDS Umbrella Grants Management (UGM) mechanisms funded by USAID/South Africa. Let me first thank you very much for participating in this first interview. We would like to learn more about your experience with both the capacity building and grants management support you have received from your UGM Umbrella Partner (Pact, FHI 360, or RTC). We want to understand how the UGM mechanism works and what effects (if any) it has had on your program(s) up to now. We will also focus on what worked the most and what worked the least about the UGM mechanism. Learning more about your experience will allow the Evaluation Team to identify areas for improvement. There are no right or wrong answers but rather differing points of views. Please feel free to share your point of view, even if it differs from what others have said. Neither our draft nor our final evaluation report will include any of your names attached to comments. You may be assured of complete confidentiality. The interview is expected to take approximately 1.5- 2 hours with an additional 30 minutes to collect and review documents. Do you have any questions before we begin? 78 South Africa Umbrella Grants Management (UGM) Performance Evaluation Section 2: Understanding Of UGM 2.1 Let’s begin by asking what first comes to mind when you hear the word UGM. 2.2 What was your understanding of the reasons for your organization being made a sub-partner of the UGM mechanism? Section 3: UMG Application Process And Pre-Award Assessment 3.1 In which Fiscal Year (between 2007 and 2011) did your organization become a UGM sub-partner? 3.2 Have you been a partner with the same UGM from that time up to the present? 3.3 Which of your programs are being funded under the UGM? 3.4 Did your organization undergo a Pre-Award Assessment from your umbrella partner at the start of your UGM partnership? ☐ Yes ☐ No ☐ Don’t know 3.5 If yes, how has your organization used the information from the organization assessment? 3.6 Were any subsequent assessments conducted (e.g., at midpoint of your grant award)? ☐ Yes ☐ No ☐ Don’t know 3.7 How you would describe your relationship with your UGM Partner? 3.8 How would you describe your relationship with the USAID Activity Manager? 3.9 If you have ever had issues with either your Activity Manager or UGM Partner, what were they, and how did you deal with them? Section 4: Governance and Leadership 4.1 How often did your Board meet last year? (the value associated with response needs to be reported in the empty box) It never met Once More than once ASK FOR THE BOARD MEETINGS’ MINUTES 4.2. Are your Board member positions time-bound? No, they are not. If they are, they last for more than 2 years Yes, they last for nearly 2 years Yes, for less than 2 years 4.3. Are processes in place within your organization to mitigate conflicts among leaders 79 South Africa Umbrella Grants Management (UGM) Performance Evaluation and staff? No, they are not. Yes, they are, but they are not documented Yes, they are and specific written guidelines are available for this purpose 4.4. What is your organizations’ mission? (compare it with what is included in the organization’s constitution) ………………………………………………………………………………………………… - ASK FOR THE CONSTITUTION The response provided is different than what is stated in the constitution Response provided is Quasi-identical to what is stated in the constitution The response provided is Identical to what is stated in the constitution 4.5. To what extent did the UGM help you enhance the transparency of decision-making within your organization? No specific Guidelines on decision-makings exist Guidelines on decision-making processes exist but not everybody is aware of them Guidelines on decision-making exist and everybody is aware of them 4.6 Are staff job descriptions detailing Key Performance Areas available? No Yes, for some staff members but not all of them Yes, for all the staff members ASK SUB-PARTNER FOR A COPY OF THE JOB DESCRIPTIONS (NO SALARY INFORMATION) FOR ALL STAFF MEMBERS Section 5: Financial and Grants Management 5.1. How timely has the disbursement of UGM funds been to your organization? VERY TIMELY TIMELY NEUTRAL NOT TIMELY NOT AT ALL TIMELY O O O O O Please describe: 5.2 It often happens that an organization takes some time to understand how a grants-making mechanism works. As a result, it may happen that a UGM sub-partner is not always able to comply with the UGM guidelines. In the case of your organization, did this happen? 80 South Africa Umbrella Grants Management (UGM) Performance Evaluation 5.3 If yes, how many times? (Check for documents - field visits reports, audits or reassessments) 5.4 To what extent were you able to address any of these grants disbursement-related issues (instance of noncompliance)? Please explain: We were not able to address or it took us more than 4 weeks from the notification of non- compliance It took us 3-4 weeks from the notification of non-compliance It took us less than 2 weeks from the notification of non-compliance 5.5 Does (UGM partner) make site visits to your organization? ☐Yes ☐No (skip to 5.9) ☐ Don’t know (skip to 5.9) 5.6. If yes, how often? 5.7 If yes, who participates in these site visits? 5.8 If yes, what is the benefit of these site visits to your organization? 5.9. To what extent were your disbursements consistent with your implementation plans last year? Not consistent Consistent most of the time Consistent all the time 5.10 Did you experience any delay in submitting your financial report to the UGM? Yes, more than twice Yes, only once Never 5.11 Did you overspend (did you spend more than what you had in your budget on a yearly basis)? Yes, more than twice Yes, only once Never Section 6: Capacity Building 6.1. What does Capacity Building mean to you? 6.2 What is your understanding of the UGM partner’s Capacity Building model? 6.3. Has (UGM partner) supported your organization to develop a capacity building plan? ☐ Yes ☐ No ☐ Don’t know 81 South Africa Umbrella Grants Management (UGM) Performance Evaluation a) If yes, did your organization play an active role in designing this plan? ☐ Yes ☐ No ☐ Don’t know b) Did this plan address any of the gaps identified in the pre-award/subsequent assessments? 6.4. What is your level of progress in relation to this plan? 6.5. Has your UGM Umbrella Partner provided capacity building to your organization in: CATEGORY YES NO DON’T KNOW Data Management Data Quality Assurance Monitoring and evaluation Organizational Governance (e.g., leadership, separation of governance and management functions, governance systems) Strategic Planning Financial and Grants Management Resource Mobilization for Sustainability Program design and implementation (for service delivery) 6.6 What capacity building support, if any, did you receive for the development of detailed annual work plans and Budgets for the grant you received from the UGM partner? a) What were the key lessons learned through this process? b) Were you able to transfer learning from this annual work plan and budget development process to the management of any other funding your organization receives? (Note: this could be an indicator for increased/built capacity) 6.7 What was the most common method for receiving capacity building support within the scope of the UGM mechanism? (Tick one) Trainings for multiple partners Customized, onsite training Mentoring through Site Visits Mentoring through telephones, emails Other 6.8 What was the most effective method for receiving capacity building support within the scope of the UGM mechanism? (tick one) Trainings for multiple partners Customized, onsite training Mentoring through Site Visits 82 South Africa Umbrella Grants Management (UGM) Performance Evaluation Mentoring through telephones, emails Other 6.9 Overall, to what degree do you believe that the capacity building support you received from (UGM partner) have met your organization’s needs for capacity building? Explain. VERY MUCH MUCH NEUTRAL LITTLE VERY LITTLE O O O O O 6.10 Have there been opportunities for information sharing and cross learning between your organization and other UGM sub partners? ☐ Yes ☐ No ☐ Don’t know a) If so, how has this occurred? b) If so, can you give us an example or examples of cross learning that you have received from other sub grantee partners or that your organization has provided to other sub grantee partners? 6.11 Are other donors providing capacity buildings support for the same programs supported by the UGM partner? 6.12 If so, what donors and what capacity building technical support activities do they provide? Section 7: External Technical Support 7.1. Did your organization receive any technical support from external consultants as part of the UGM partnership? ☐Yes ☐ No ☐ Don’t know 7.2 If yes, please describe (explore how this support relates to the overall capacity building plan) Section 8: Service Delivery 8.1 Has the UGM partnership helped your organization to meet or exceed program service delivery objectives (please specify is this was due to USAID or the UGM partner)? ☐ Yes ☐ No ☐ Don’t know Please explain. 8.2. As a result of the support provided to you by the UGM/Service Provider, have you started offering a new service (please specify is this was due to USAID or the UGM partner)? No Yes, one new Yes, more than one 83 South Africa Umbrella Grants Management (UGM) Performance Evaluation 8.3. As a result of the support provided to you by the UGM/Service Provider, are you targeting new population groups (please specify is this was due to USAID or the UGM partner)? No Yes, one new Yes, more than one 8.4 As a result of the support provided to you by the UGM/Service Provider, are you now targeting new geographical areas (please specify is this was due to USAID or the UGM partner)? No Yes, one new Yes, more than one 8.5. Question: As a result of the support provided to you by the UGM/Service Provider, has the quality of your services changed(please specify is this was due to USAID or the UGM partner)? No Yes, partly Yes, very much 8.6. Was the implementation of your activities ever interrupted? Yes, for more than two weeks Yes, for one to two weeks Never 8.7 Have there been any unintended impacts for your beneficiaries as a result of your organization’s UGM partnership? ☐ Yes ☐ No ☐ Don’t know 8.7 If yes, please describe. Indicate if the impacts or positive, negative, or both. Section 9: Defining and Measuring Success 9.1. What indicators are you monitoring for your programs funded by the UGM? (Please provide a copy of a recent progress report showing what indicators are monitored/reported) 9.2 What indicators are you monitoring to measure capacity building/development in your organization? 9.3 What assistance/capacity building support did the UGM provide your organization to monitor and evaluate its services in relation to these indicators? 9.4 Please share with us an example of visible capacity increase within your organization. 84 South Africa Umbrella Grants Management (UGM) Performance Evaluation Section 10: Sustainability 10.1. Has your organization been successful in securing any new funding as a result of UGM resource mobilization support? ☐Yes ☐ No ☐ Don’t know 10.2. If so, please describe. 10.3. Are your policies/activities aligned with relevant national government plans and policies? No, or, if yes, a few of them Yes, most of them are Yes, all of them are 10.4 Is your organization sharing the costs of the project funded by the UGM? No (skip to 10.6) Yes, for most of the costs (see 10.5) Yes, for all of the costs )see 10.5) 10.5. If yes, how so? National Government subsidies Beneficiaries contributions Other funds (please describe) 10.6. If you look at your organization’s overall budget, how much do UGM funds account for (in percentage terms?) More than 70% 50–50% Less than 70% 10.7. Has your organization established written guidelines or policies to guide operations in following areas: CATEGORY Written guidelines exist <2 years Written guidelines exist for 2 years Written guidelines exist >2 years Financial Management HR Management 85 South Africa Umbrella Grants Management (UGM) Performance Evaluation Monitoring and evaluation Strategic Plan Work Plan (current year) 10.8. Does your organization have written technical guidelines for your programs, and/or training guidelines for the staff who deliver the services? No, it doesn’t have any such written technical guidelines Yes, one of the its guidelines (programmatic or training) is written and in current use Yes, more than one of its guidelines (programmatic or training) is written and in current use. Section 11: Innovation 11.1. Can you describe any significant innovations that have emerged from the capacity building support provided by the UGM? 11.2. Has your organization identified any Best Practices that you plan to continue as part of your organizational internal management or service delivery going forward? ☐ Yes ☐ No ☐ Don’t know If yes, please explain. Section 12: Suggestions For Enhancing UGM Responsiveness and Agility 12.1 Based on your experience, what would be a good strategy to put in place to make the current UGM structure more responsive, faster, or more flexible? 12.2 Is there anything else you’d like to share with us? Documents NEEDED: • Copy of the contracts/Terms of Reference for all SUB-PARTNERS staff members • SUB-PARTNER’S BOARD MEETINGS MINUTES • LAST PROGRESS REPORT AND M&E PLAN • SUB-PARTNER’S CONSTITUTION THANK YOU VERY MUCH FOR YOUR TIME. IF YOU HAVE ANY QUESTIONS ABOUT THE EVALUATION PROCESS, OR WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE FEEL FREE TO COMMENT NOW, OR WRITE TO US AT THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM (provided this email) 86 South Africa Umbrella Grants Management (UGM) Performance Evaluation AGAIN, ALL YOUR COMMENTS AND E-MAILS WITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT APPEAR IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). SCORING MATRIX For each of the four matrix dimensions below, please report the corresponding average numerical score. So, for example, let's say that you are administering the questionnaire for the UGM sub-partners and you sum all the values plugged in the 6 blank boxes under the Governance and Leadership Section. If your final score is 12, then you will just need to divide 12 by 6 and your final score will be2. Just report this number for now. We will discuss the interpretation of the different averages next week. 87 South Africa Umbrella Grants Management (UGM) Performance Evaluation FHI 360 CAPABLE PARTNERS QUESTIONNAIRES 4A. Questionnaire for CAP Staff Follow-Up Questions for Fhi360 Capable Partners Staff First Section: Capacity Building Provided To DOH Partners and Local Communities 1.1 What do you understand by capacity building? 1.2 At the CAP national strategic planning workshops, NDOH and PDOHs present their training needs: a) How does CAP decide which training needs to respond to? b) How does CAP decide which provinces to provide training in? c) How does CAP plan the scheduling of training dates with the provinces? 1.3 Does CAP communicate the national training plan to NDOH and PDOH i.e. do national and provincial DOHs know how many and what type of trainings are happening in the different provinces? a) When does CAP communicate this information? b) How does CAP communicate this information? 1.4 How is the CAP national strategic planning process aligned with national DOH strategic planning for an integrated response to PMTCT? 1.5 What training materials are provided to DOH co-facilitators and learners during the various training programs? (Please indicate with a tick for each course where a facilitator guide, learner guide, PowerPoint slides or other materials are provided) 88 South Africa Umbrella Grants Management (UGM) Performance Evaluation Training course Facilitator Guide Learner Guide Powerpoint Slides Other 1-day Decision Makers Course 10-day Integrated PMTCT Training (Core Course) 5-day Abridged Integrated PMTCT Training 3-day Abridged Integrated PMTCT Training for Doctors/ Sr. Managers 3-day Abridged Integrated PMTCT Training for Implementers 5-day BFHI Assessors Training Code Workshop 5-day Social and Behavior Change Communication (SBCC) Course On-Site Mentoring and Coaching 1-day Workshop for Strengthening Clinic Committee 1.6 Does CAP have other training materials may have been developed for other countries that are relevant for PMTCT but that have not been incorporated into the South African training programs? Please explain. 1.7 Has CAP ever funded external technical support through consultants to national/provincial DOH? If yes, what kind of external technical support was funded? If no, does this mean that CAP always has the internal technical capacity to respond to emerging needs e.g. family planning; male involvement in PMTCT? 1.8 How does CAP view their reporting obligations to national DOH? a) What kind of information does CAP report? b) To whom is CAP reporting at this level? a) How often does CAP report? b) How does CAP report? 1.9 How does CAP view their reporting obligations to provincial DOH? a) What kind of information does CAP report? b) To whom is CAP reporting at this level? c) How often does CAP report? d) How does CAP report? 89 South Africa Umbrella Grants Management (UGM) Performance Evaluation 1.10 How does CAP ensure that best practices and lessons learned at district/sub-district level are shared between provincial DOHs and with national DOH stakeholders? 1.11 For which provinces does CAP have a sign MOU with the provincial DOH? (We noted one MOU with North West in the documentation provided)? a) What is the purpose of this MOU? b) How necessary is this MOU for CAP capacity building work? c) Is USAID involved in negotiations around the drafting and signing of these MOUs? Second Section: Provincial and District Capacity to Integrate PMTCT Services 2.1 From the documentation it appears that rapid assessments were undertaken by AED/FHI CAP before training was undertaken in communities. There is also one evaluation which compared knowledge, attitudes and practices with previous baseline. a) Were any changes to the CAP capacity building approach/curriculum based on the findings? b) Does CAP have plans to do follow-up evaluations to compare knowledge, attitudes, and practices against the baseline findings? DEAR CAP TEAM, THANK YOU FOR COMPLETING THE SURVEY. IF YOU WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE WRITE TO THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM ALL YOUR E-MAIL EXCHANGES WITH THE SOCIAL IMPACT EVALUATION TEAMWILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT BE MENTIONED IN DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 90 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4B. Questionnaire for National Directorate Of Health (NDOH) Cap Evaluation Questions for National DOH Stakeholders Working With FHI 360 Capable Partners Department/Directorate: _________________________________________________ Respondent Title & Names: ______________________________________________ Location: _____________________________________________________________ Date/Time: ____________________________________________________________ Introduction: My name is__________________________ and I am a member of the Social Impact Evaluation Team currently assessing the performance of the HIV/AIDS Umbrella Grants Management (CAP) mechanisms funded by USAID/Southern Africa, which includes FHI360 Capable Partner. We received your name from the CAP staff in Pretoria and they suggested that we meet with you to discuss the involvement of your national Department of Health/Directorate in the activities/programs funded by CAP. To this end, the FHI 360 Capable Partners communicated with you last week requesting that you kindly meet with us. On behalf of my evaluation team, let me first thank you very much for accepting to participate in this first interview. Then, please allow me to clarify the objective of our meeting today: as part of this evaluation, we would like to learn more about your experience with technical capacity building support you have been receiving from FHI 360 Capable Partners over the past few years. What we are particularly interested is to gain a better understanding of how the CAP mechanism works and what effects (if any) it has had on your program(s) up to now. Our exchange today will also focus on what worked the most and what worked the least about the CAP mechanism. Any comment or idea you will share with us in the course of this interview will be kept confidential and anonymous: neither our draft nor our final evaluation report will include your name attached to comments. Please also keep in mind that we’re interested in all comments positive and negative. The interview is expected to take approximately 1–1 ½ hours. Do you have any questions before we begin? Let’s begin by asking what your Directorates role is in supporting provinces to prepare health workers at the clinic sites and in the community to integrate PMTCT and Nutrition? Section 2: Strategic Planning The following questions focus on the national DOH role in the annual CAP national strategic planning to build technical capacity in DOH 2.1 Did your Directorate participate in the national strategic planning process that CAP facilitated in 2011? Yes (Go to Q 2.2) No (Skip to Section 3) Don’t know (Skip to Section 3) 2.2 Please explain what you understand to be the objectives of this national strategic planning process. 91 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2.3 What is the relationship between the strategic capacity building plan you develop for your Directorate and the national strategic planning process facilitated by CAP? 2.4 How would you rate the national strategic planning process that your Directorate participated in with CAP in 2011? VERY GOOD GOOD AVERAGE BELOW AVERAGE POOR O O O O O Please explain: __________________________________________________________ 2.5 What has worked best about the CAP national strategic planning? Please list the three most relevant aspects of the planning process that come to your mind 2.6 What would you change (if anything) about the CAP national strategic planning process? Section 3: Capacity Building 3.1 What is your understanding of the FHI 360 capacity building model? 3.2 What are the key strengths of this model? 3.3 Is there anything about this model that you think could be improved? 3.4 What role, if any, did FHI 360 CAPABLE Partners play in support your national Directorate to develop curriculum materials for the integration of PMTCT and nutrition? ☐ Yes ☐ No ☐ Don’t know ☐ Not applicable 3.6. What role, if any, did your Directorate play in developing these materials? 92 South Africa Umbrella Grants Management (UGM) Performance Evaluation 3.7 Do you know if the provincial Directorate formally endorsed or accredited (or approved) any of the training materials used by CAP? Do you know if any of the courses be used for CPD points? 3.8 Have staff in your national Directorate been trained in any of these curricula? How is the coaching/mentoring aspect applied at a national level? 3.9 Does your National Directorate conduct any joint site visits with CAPABLE Partners staff to go see where some of the capacity building activities are implemented? If so, do you participate directly in these visits? What is your role in these visits? 3.10 In how many courses provided by FHI360 Capable Partners did the national DoH play a co-trainer role? 3.11 Of the capacity building support you have received from FHI360 CAP, which aspect was the most valuable Why? 3.12 Of the capacity building support you have received from FHI360 CAP, what aspect was the least valuable? Why? 3.13 What are some of the unmet capacity building (service delivery) needs related to the integration of PMTC and Nutrition at national, provincial and site level? 3.15 What capacity-building could be provided in the future to strengthen DoH regional training centers? Section 4: External Technical Support Formally endorsed Accredited CPD points Training course Yes No Don’t know n/a Yes No Don’t know n/a Yes No Don’t know n/a 1-day Decision Makers Course 10-day Integrated PMTCT Training (Core Course) 5-day Abridged Integrated PMTCT Training 3-day Abridged Integrated PMTCT Training for Doctors/ Sr. Managers 3-day Abridged Integrated PMTCT Training for Implementers 5-day BFHI Assessors Training Code Workshop 5-day Social and Behavior Change Communication (SBCC) Course On-Site Mentoring and Coaching 1-day Workshop for Strengthening Clinic Committee 93 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4.1 As part of the CAP partnership, have you received external technical support from? (E.g. consultants) If yes, for what purpose? In which area? 4.2 What are the key outcomes or effects resulting from the Technical Support received from the CAP? Section 5: Service Delivery 5.1 How have the Trainings provided by FHI360 CAPABLE Partners prepared health workers at the clinic sites and in the community to integrate PMTCT and Nutrition? 5.2 Has your Directorate benefited from other capacity building initiatives or support provided by other donors that could have affected the current level of integration of PMTCT or nutrition services? If yes, were such programs any different than the ones provided by FHI 360 Capable Partners. If yes, how so? 5.3 Also, how would you differentiate the effect of CAP activities and the effect of activities funded by other donors? Please explain below. Section 6: Monitoring and Evaluation 6.1 Does your Directorate have comprehensive Monitoring and Evaluation system for tracking progress in provinces and sites? ☐Yes ☐ No ☐ Don’t know 6.2 What was CAPs role, if any, in developing this system? 6.3 What indicators do you use, if any, to measure increased capacity in service delivery? Section 7: Sustainability 7.1 Has your department taken steps/made progress in assuming leadership (delivering training; funding) for any of the training being delivered by CAP? If so, please describe. a) Delivering training? b) Cost sharing? Section 8: Communication And Coordination 8.1 How has FHI360 CAP communicated and coordinated with your Directorate? What worked best, and what would you do differently? Any last comments/thoughts? 94 South Africa Umbrella Grants Management (UGM) Performance Evaluation THANK YOU VERY MUCH FOR YOUR TIME. IF YOU HAVE ANY QUESTIONS ABOUT THE EVALUATION PROCESS, OR WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE CAP PERFORMANCE EVALUATION, PLEASE FEEL FREE TO WRITE TO US AT THE FOLLOWING ADDRESS: CAP.EVALUATION@GMAIL.COM AGAIN, ALL YOUR COMMENTS AND E-MAILSWITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT APPEAR IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 95 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4C. Questionnaire for Provincial Directorate of Health (PDOH) UGM Evaluation Questions For Provincial DOH Stakeholders Working With FHI 360 Capable Partners Department/Directorate: _________________________________________________ Respondent Title & Names: ______________________________________________ Location: _____________________________________________________________ Date/Time: ____________________________________________________________ Introduction: My name is__________________________ and I am a member of the Social Impact Evaluation Team currently assessing the performance of the HIV/AIDS Umbrella Grants Management (UGM) mechanisms funded by USAID/Southern Africa, which includes FHI360 Capable Partner. We received your name from the CAP staff in Pretoria and they strongly suggested that we meet with you to discuss the involvement of your provincial DoH Department/Directorate in the activities/programs funded by UGM. To this end, the FHI 360 Capable Partners communicated with you last week requesting that you kindly meet with us. On behalf of my evaluation team, let me first thank you very much for accepting to participate in this first interview. Then, please allow me to clarify the objective of our meeting today: as part of this evaluation, we would like to learn more about your experience with both the capacity building and grants management support you have been receiving from FHI 360 Capable Partners over the past few years. What we are particularly interested is to gain a better understanding of how the UGM mechanism works and what effects (if any) it has had on your program(s) up to now. Our exchange today will also focus on what worked the most and what worked the least about the UGM mechanism. Before we start, let me remind you of some of our procedures. Also, any comment or idea you will share with us in the course of this interview will be kept confidential and anonymous: neither our draft nor our final evaluation report will include your name attached to comments. Please also keep in mind that we’re just as interested in negative comments as in positive ones, and at times, the negative comments are the most helpful. The interview is expected to take approximately 1 – 1 ½ hours. Do you have any questions before we begin? Let’s begin by asking what your Directorates role is in supporting provinces to prepare health workers at the clinic sites and in the community to integrate PMTCT and Nutrition? Section 2: Strategic Planning for Capacity Building The following questions focus on the provincial DOH role in the annual CAP national strategic planning to build technical capacity in DOH: 2.7 Did your Directorate participate in the national strategic planning process that CAP facilitated in 2011? 2.8 Please explain what you understand to be the objectives of this national strategic planning process. 2.9 What is the relationship between the strategic capacity building plan you develop for your Directorate and the national strategic planning process facilitated by CAP? 96 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2.10 During this process did CAP make you aware of any training materials apart from nutrition that related to PMTCT that you were not already aware of? 2.11 What capacity building support did you ask CAP in your province? 2.12 In which areas (Districts/sub-districts) is CAP providing this support? 2.13 How would you rate the national strategic planning process that your province participated in with CAP in 2011? 2.14 How would you rate the follow up conducted by CAP on the training plans made in the last year? 2.15 What has worked best about the CAP national strategic planning? Please list the three most relevant aspects of the planning process that come to your mind 2.16 What would you change about the CAP national strategic planning process? Section 3: Capacity Building 3.1 What is your understanding of the FHI 360 capacity building model? 3.2 What was the most COMMON/FREQUENT modality for receiving capacity building support from FHI360 CAP? 3.3 What was the most EFFECTIVE modality for receiving capacity building support from FHI360 CAP? 3.4 Does your Provincial Directorate conduct any joint site visits with CAPABLE Partners staff to go see where some of the capacity building activities are implemented? 3.5 Can you tell us more about the trainings for community participants? a) Which of the various trainings do community members take part in? b) Who in the community would normally participate? c) Who selects/invites the community members? 3.5 In how many courses provided by FHI360 Capable Partners did the provincial DoH play a co-trainer role? (Please indicate the number as closely as you can) 3.6 Of the capacity building support you have received from FHI360 CAP, which aspect was the most valuable Why? 3.7 Of the capacity building support you have received from FHI360 CAP, what aspect was the least valuable? Why? 3.8 Overall, to what degree do you believe that the capacity building support you received from CAP have met your Districts needs for capacity building? 3.9 Please explain progress on cost sharing with government – how or to what extent has DOH participated in the cost of training at provincial and district/sub-district level? 3.10 What capacity-building could be provided in the future to strengthen DoH regional training centers? 3.11 What are some of your unmet capacity building (service delivery) needs related to the integration of PMTC and Nutrition? 97 South Africa Umbrella Grants Management (UGM) Performance Evaluation Section 4: External Technical Support 4.1 As part of the CAP partnership, have you received external technical support from? (E.g. consultant came to work with you on a particular area) If yes, for what purpose? In which area? 4.2 What are the key outcomes or effects resulting from the Technical Support received from the UGM? 4.3 What are some of your unmet technical support (service delivery) needs? Section 5: Service Delivery 5.1 How have the Trainings provided by FHI360 CAPABLE Partners prepared health workers at the clinic sites and in the community to integrate PMTCT and Nutrition? 5.2 Which one(s) of your service delivery activities benefited from CAP support the most? 5.3 Which one(s) of your service delivery activities benefited from CAP support the least? 5.4 How has the UGM affected the coverage of your activities in terms of: a) More people targeted? b) Larger range of activities implemented? c) Large geographic scope? 5.5 Have there been any unintended impacts for your clients as a result of the partnership between your Directorate and FHI 360 CAP? (Any surprises) 5.6 Has your district benefited from other capacity building initiatives or support provided by other donors that could have affected the current level of integration of PMTCT or nutrition services? If yes, were such programs any different than the ones provided by FHI 360 Capable Partners? If yes, how so? 5.7 Also, how would you differentiate the effect of CAP activities and the effect of activities funded by other donors? Please explain below. Section 6: Monitoring and Evaluation 6.1 Has FHI360 CAP assisted your Directorate to develop a comprehensive Monitoring and Evaluation system for tracking progress? Section 7: Sustainability 7.1 Has your department taken steps/made progress in assuming leadership (delivering training; funding) for any of the training being delivered by CAP? If so, please describe. a. Delivering training? b. Cost sharing? Section 8: Communication and Coordination 8.1 How has FHI360 CAP communicated and coordinated with you? What worked best, and what would you do differently? (On-going communication process?) 98 South Africa Umbrella Grants Management (UGM) Performance Evaluation Any last comments/thoughts? THANK YOU VERY MUCH FOR YOUR TIME. IF YOU HAVE ANY QUESTIONS ABOUT THE EVALUATION PROCESS, OR WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE FEEL FREE TO WRITE TO US AT THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM AGAIN, ALL YOUR COMMENTS AND E-MAILSWITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT APPEAR IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 99 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4D. CAP ONLINE SURVEY CAP Online Survey Introduction: The evaluation team for the USAID South Africa UGM mechanism seeks to learn about the partnership between FHI Capable Partners and provincial departments of health (DOH) in Free State, North West, Limpopo and Western Cape for provision of technical assistance to improve the quality of PMTCT services. We would very much appreciate your taking the time to fill out this brief survey, which will help us to define effective interview tools for follow-up interviews with provincial department of health stakeholders and health facility staff. First Section: Overall Understanding of UGM Objectives and Processes 1.1 When your provincial department of health first became involved in the technical assistance (TA) partnership with FHI 360 Capable Partners to improve PMTCT services, which documents did you and your colleagues receive that helped to define the objectives and expectations of this partnership? (Please rank the documents in order of usefulness) 1. 2. 3. 4. _ None 1.2. Did these documents (if any) clarify an expected transition of TA from FHI 360 Capable Partners to the provincial health departments (DOH) as part of the USAID Partnership Framework mandate that emphasizes country ownership, technical support to improve the South African Government capacity and long term sustainability of programs that have received PEPFAR support? ☐ Yes ☐ No ☐ Don’t know ☐ Not applicable 1.3 How clear did your provincial department of health find the variety and level of PMTCT related capacity building activities to be undertaken in partnership with FHI 360 Capable Partners? VERY CLEAR CLEAR NEUTRAL UNCLEAR VERY UNCLEAR DO NOT KNOW O O O O O O 100 South Africa Umbrella Grants Management (UGM) Performance Evaluation Second Section: Strengthened Capacity At PMTCT Sites 2.1 From your perspective, how well has the FHI360 Capable Partners’ PMTCT Training of Trainers (TOT) course designed for auxiliary nurses and lay counselors to equip them with appropriate PMTCT knowledge and skills adequately prepared health workers at the PMTCT sites in your province? VERY WELL WELL NEUTRAL POORLY VERY POORLY DO NOT KNOW O O O O O O 2.2 Which of the following have been included in the TA provided for PMTCT sites in your province? (Please check all that apply) ☐ Pre-service training courses for auxiliary nurses and lay counselors ☐ In-service training courses for auxiliary nurses and lay counselors ☐ Collaboration between DOH and FHI360 Capable Partners in designing the TA to ensure activities fit into the existing health system to help promote sustainability ☐ Clarification of performance expectations for newly trained staff and managers ☐ Strengthened supportive supervision processes ☐ Strengthened referral systems to enhance continuity of care ☐ Improved functional referrals from PMTCT to FP services in order to promote healthy spacing of pregnancies and prevent unintended pregnancies among post-partum PMTCT clients ☐ Training on couple counseling and strategies to involve male partners in PMTCT visits ☐ Development of FP messages to be incorporated into points in the service delivery system that have shown to increase the likelihood of uptake of FP (e.g., pre-/post-test counseling, post-partum period, infant feeding counseling, infant testing, or child health services) 2.3 From your perspective, how much has provision of ARV prophylaxis at PMTCT sites increased as a result of the TA provided by the UGM partnership between FHI 360 Capable Partners and the provincial department of health? VERY MUCH MUCH SOMEWHAT LITTLE VERY LITTLE DO NOT KNOW O O O O O O 2.4 From your perspective, how much has the provision of FP counseling and referrals increased as a result of the TA provided by the UGM partnership between FHI 360 Capable Partners and the provincial department of health? VERY MUCH MUCH SOMEWHAT LITTLE VERY LITTLE DO NOT KNOW O O O O O O 101 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2.5 Overall, how much do you believe that services at PMTCT sites in your province have been improved as a result of the UGM TA partnership with FHI 360 Capable Partners? VERY MUCH MUCH SOMEWHAT LITTLE VERY LITTLE DO NOT KNOW O O O O O O Third Section: Strengthened Capacity Of Provincial Departments Of Health 3.1 From your perspective, how much has the UGM TA partnership with FHI360 Capable Partners strengthened the provincial health department’s capacity to train and support health workers at PMTCT sites? VERY MUCH MUCH SOMEWHAT LITTLE VERY LITTLE DO NOT KNOW O O O O O O 3.2 Please indicate the areas where institutional capacity at the provincial department of health level has improved the MOST. (Please check all that apply) ☐ Pre-service training courses for auxiliary nurses and lay counselors ☐ In-service training courses for auxiliary nurses and lay counselors ☐ Ensuring capacity-building activities fit into the existing health system ☐ Clarification of performance expectations for newly trained staff and managers ☐ Strengthened supportive supervision processes ☐ Strengthened referral systems to enhance continuity of care ☐ Improved functional referrals from PMTCT to FP services ☐ Training on couple counseling and strategies to involve male partners in PMTCT visits ☐ Development of FP messages to be incorporated into points in the service delivery system that have shown to increase the likelihood of uptake of FP ☐ None of the above ☐ Don’t know Please feel free to explain your selection further. 3.3 Likewise, please indicate the areas where institutional capacity at the provincial health department level has improved the LEAST. (Please check all that apply) ☐ Pre-service training courses for auxiliary nurses and lay counselors ☐ In-service training courses for auxiliary nurses and lay counselors ☐ Ensuring capacity bilding activities fit into the existing health system ☐ Clarification of performance expectations for newly trained staff and managers ☐ Strengthened supportive supervision processes 102 South Africa Umbrella Grants Management (UGM) Performance Evaluation ☐ Strengthened referral systems to enhance continuity of care ☐ Improved functional referrals from PMTCT to FP services ☐ Training on couple counseling and strategies to involve male partners in PMTCT visits ☐ Development of FP messages to be incorporated into points in the service delivery system that have shown to increase the likelihood of uptake of FP ☐ None of the above ☐ Don’t know Please feel free to explain your selection further. 3.4 With regard to the UGM Technical Assistance for improved PMTCT services, how often has your department communicated with the following partners? a) FHI360 Capable Partners VERY OFTEN OFTEN RARELY VERY RARELY DO NOT KNOW O O O O O Please provide any suggestions for improving communication between your organization and FHI360 Capable Partners. b) USAID VERY OFTEN OFTEN RARELY VERY RARELY DO NOT KNOW O O O O O Please provide any suggestions for improving communication between your organization and USAID c) Other participating provincial departments of health VERY OFTEN OFTEN RARELY VERY RARELY DO NOT KNOW O O O O O Please provide any suggestions for improving communication between your organization and other provincial department of health partners , if you think this would be beneficial. 3.5 To your knowledge, is a written, phased plan in place for the transition of ownership and leadership of PMTCT related TA from the UGM/FHI360 CP to the provincial department of health or to the national department of health? ☐ Yes ☐ No ☐ Don’t know 103 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4.E. FHI360 (CAP) SITE VISIT QUESTIONNAIRE Questions for FHI360 Capable Partners Site Visit Site Name: ______________________________________________________________ Date: _________________________________________________________________ Time: _________________________________________________________________ Names/titles of Interviewees: ________________________________________________ 1. What trainings has FHI 360 Capable Partners (CAP) provided at this site? When? How many times? 2. How many people participated directly in these trainings? Health workers, what categories Community committee members Others 3. What do you see as the main benefits to you from having participated in training provided by CAP? Please choose which apply: ☐ Increased knowledge ☐ Increased skills ☐ Improved services ☐ Expanded services ☐ Better communication with clients ☐ Strengthened relationships between facility and community ☐ Increased community demand for services ☐ Increased male involvement in PMTCT services ☐ Improved outcomes for HIV positive mothers ☐ Improved outcomes for HIV exposed infants ☐ Other 4. How have you incorporated learning from the training into your work? or How have you changed the way you work as a result of the training? 5. Has the CAP training influenced the way in which you COLLECT and REPORT data related to PMTCT and/or nutrition at this site? If so, how? 104 South Africa Umbrella Grants Management (UGM) Performance Evaluation 6. Has the CAP training influenced the way in which you USE this data? If so, how? (ask to see data collection tools). 7. How have your clients/community workers benefited from what you learned during training? (have been told that early bookings for ANC have happened in KZN as a result of trainings – triangulation) 8. Have you received follow up visits from CAP related to the training? If so, how often? What happens during these visits? 9. Have there been any changes in supportive supervision for integrated PMTCT and nutrition apart from these visits that you would relate directly to CAP’s presence at this site? If so, please explain. 10. Have you received trainings related to PMTCT and/or nutrition from any other training organization at this site? If so, how were they similar or different from CAP trainings? 105 South Africa Umbrella Grants Management (UGM) Performance Evaluation DSD QUESTIONNAIRE UGM Evaluation Questions For National DSD Stakeholders Working With Pact Department/Directorate: ____________________________________________________ Location: _______________________________________________________________ Respondent Title & Names: _________________________________________________ Date/Time: ______________________________________________________________ Introduction: My name is__________________________ and I am a member of the Social Impact Evaluation Team currently assessing the performance of the HIV/AIDS Umbrella Grants Management (UGM) mechanisms funded by USAID/Southern Africa, which includes Pact. We received your name from the USAID mission in Pretoria and they strongly suggested that we meet with you to discuss the involvement of your Department/Directorate in the activities/programs funded by UGM. To this end, USAID communicated with you last week requesting that you kindly meet with us. On behalf of my evaluation team, let me first thank you very much for agreeing to participate in this first interview. Then, please allow me to clarify the objective of our meeting today: as part of this evaluation, we would like to learn more about your experience with both the technical assistance support you have been receiving from USAID funding via Pact over the past few years. What we are particularly interested is to gain a better understanding of how the UGM mechanism works and what effects (if any) it has had on your program(s) up to now. Our exchange today will also focus on what worked the most and what worked the least about the UGM mechanism. Before we start, let me remind you of some of our procedures. Also, any comment or idea you will share with us in the course of this interview will be kept confidential and anonymous: neither our draft nor our final evaluation report will include your name attached to comments. Please also keep in mind that we’re just as interested in negative comments as in positive ones, and at times, the negative comments are the most helpful. The interview is expected to take approximately 1 – 1 ½ hours. Do you have any questions before we begin? 1. Let’s begin by asking what first comes to mind when you hear the word Pact. 1.1.1 Accessing technical support from USAID via Pact 1.1.2 When did NDSD first access technical support from USAID via Pact? 1.2.1 Which Directorates/Sub-Directorates have accessed technical support from Pact? (e.g., HIV/AIDS; Child Protection) 1.2.2 How are priorities for external technical support identified by your Department/Directorate? a) What is USAID’s involvement in this decision-making process? b) What is Pact’s involvement in this decision-making process? c) Are provincial DSDs involved in the process at all? Explore. If not, is there a role for them? d) How effective is this process in ensuring that NDSD priority needs for technical assistance are met via this mechanism? Is there anything that could be done to improve or strengthen it? 106 South Africa Umbrella Grants Management (UGM) Performance Evaluation 1.3.1 How are requests for technical support communicated to Pact/USAID? Is there an agreed on protocol that needs to be followed? 1.3.2 How you would describe your relationship with Pact? 1.3.3 How would you describe your relationship with the USAID Activity Manager? 1.4.1 Have you received or are you currently receiving external technical assistance from other donors? If so, what is the scope of such support? (Percentage of technical support received via Pact vs. other donors) 2. Planning and scheduling 2.1 Does NDSD/relevant Directorates have an annual or multi-year workplan for technical support from USAID/Pact? a) If yes, who is involved in developing this workplan? Access a copy of this? b) If no, what approach is used to guide the planning and scheduling of this support? c) How effective is this approach to planning the technical support? Any improvements that could be made to strengthen? 3. Defining and measuring success 3.1 Thinking back on all the technical support provided to your Department/Directorate by USAID via Pact which activities would you say made the most difference/impact in terms of: a) Contribution to capacity building of National DSD staff b) Contribution to capacity building of Provincial DSD staff c) Improvements in quality of service delivery d) Improvements in coverage of service delivery e) Diversified/increased funding sources (e.g. Treasury, donors) f) Translation of policy into practice g) Agility – flexibility - rapidity 3.2 What process is followed in contracting external assistance? What role does Pact play and what role does your Department/Directorate play? a) Drafting of terms of reference? b) Identification of suitable service providers? c) Review of applications/responses to TORs? d) Contract negotiations? 3.3 What works well with the current process? Is there anything that could be improved? 3.4 How timely has Pact been in contracting external technical assistance? a) Average length of time from submission of the request to signing of the contract? b) Is this a realistic response time in your view? Anything that could be done to speed it up? 3.5 Based on your experience, what would be a good strategy to put in place to simplify the current Pact /UGM structure? 3.6 What would be other strategies that it would be worth pursuing in order to enhance Pact/UGM effectiveness? 107 South Africa Umbrella Grants Management (UGM) Performance Evaluation 4. Coordination and Communication with PACT 4.1 How has Pact communicated and coordinated with your Directorate DoH? What worked best, and what would you do differently? 5. Pact Grant Making and Capacity Building Support to NPO Sector (not sure if this is relevant if interview is only focusing on technical support to NDSD) 5.1 Does the NDSD/relevant Directorate play a role in identifying priorities for USAID funding via Pact for NGO sub-partners? Explore. 5.2 As a national Department/Directorate what are your views on the relevance of a USAID UGM mechanism like Pact to provide capacity building support to the NPO s working in the OVC and HIV/AIDS care and support sectors? 5.3 From your perspective what are the key capacity building gaps that a UGM mechanism should be focusing on in the NPO sector? 6. Final reflections 6.1 What aspects of the UGM mechanism have you found most useful for your Department/Directorate? 6.2 What aspects of the UGM mechanism have you found least useful for your Department/Directorate? 6.3 Any last comments? 108 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM FOLLOW-UP QUESTIONNAIRE USAID/South Africa UGM Evaluation Follow-up Questionnaire for UGM Partners Dear UGM Partner, The Social Impact Evaluation Team currently assessing the performance of the USAID/South Africa UGM mechanism would like to ask you and your colleagues some follow-up questions. We appreciate your taking the time to respond! The following set of questions might want to be responded by the UGM Chief of Party or Project Director and key staff (please identify each respondent) Name: _____________________________________________________________ Title: _______________________________________________________________ Email address: _________________________________________________________ Phone number: _______________________________________________________ Section 1: Project Management 1.1. Can you explain to us one more time: a) When and how your organization obtained a UGM award? b) How your organization went about the process of initiating work with various sub￾grantees? 1.2. Can you remember when you applied for a UGM award/and or when you renewed it (if applicable)? Please describe the process briefly for us 1.3. Can you tell us how long it took from time of proposal to time of award? 1.4. Can you tell us, if possible, how long it took from award notification to receipt of funding in the first year? 1.5. Can you tell us if the funding cycle is regular each year? What determines when you receive funding? How do you manage if there are delays or variations in the cycle of in funding? 1.6. Your APS requires certain initial approvals from USAID as well as annual approvals. Can you describe the approvals that were/are required, and the process for obtaining them? 1.7. Can you tell us how long it took to obtain approvals? (This can be stated in terms of the time and/or effort required to obtain each of the approvals) Follow up questions about approvals (address each of the following if possible): 1.8. Can you tell us about the process of getting approval of Key Personnel? 109 South Africa Umbrella Grants Management (UGM) Performance Evaluation 1.9. Can you tell us about the process of getting approval of Work plan(s)? 1.10.Can you tell us how you handle other PEPFAR or USAID requirements: a) Semi-annual Reporting? Other reporting? b) Annual COP submissions? 1.11.UGM Project and Sub-grantee start up: delays, difficulties, or setbacks. a) Can you tell us how long the initial start-up for your organization took? b) Can you tell us what start-up tasks were required (i.e., office rental - include main office and any sub-offices), vehicle and equipment procurement, recruiting, hiring and training staff, and so on) and how long overall it took for these start-up tasks to be completed in order for your UGM to be fully operational? c) Since you became fully operational, have you remained at full force since then? d) Or, has the project experienced any major delays, difficulties, or setbacks? If so, can you tell us what they were? How did you respond to them? e) Can you tell us how long it usually takes for sub-grant-funded activities to start up and to become fully operational? f) Which sub-grantees took longer than average, which were faster than average, and why was that? g) Once the sub-grants became fully operational, have they remained at full force, or have there been delays or setbacks with some partners? Can you explain? 1.12.How closely would you say the UGM mechanism has followed the UGM objectives as proposed at the beginning of this program? VERY CLOSELY CLOSELY SOMEWHAT NOT VERY CLOSELY NOT AT ALL CLOSELY MAJOR CHANGES OCCURRED O O O O O O 1.13.If goals and objectives were fulfilled over time, can you describe what changed, when, and why? Were the changes initiated by you, or requested by USAID, or the grantee(s)? 1.14.Have the expected results remained the same or changed since the beginning? TOTALLY CHANGED MAJOR CHANGES A GOOD BIT CHANGED SLIGHTLY UNCHANGED O O O O O 1.15.For any important changes, what challenges did the changes present for implementation of the UGM? 110 South Africa Umbrella Grants Management (UGM) Performance Evaluation 1.16.Regarding the objectives and desired results, can we discuss those which you indicated were easiest to achieve so far, and which have been particularly challenging? (Refer to initial questionnaire) 1.17.USAID assessed, selected, and assigned sub-grantees to your UGM. USAID also decided the amount of funding for each partner annually. This is an interesting approach. Can you explain: a)What challenges this has presented for your organization (i.e., in terms of finances, reporting, etc.)? How you handled them from the start up to now? b)Based on what you have learned, what would you do differently if you could? c)Is there anything you would ask USAID to do differently if you could? Why? 1.18.Can we further explore the USAID financial management expectations between your organization and USAID, and between your organization and sub-grantees since the beginning of the program? (What issues if any arose?) a) What have you found most challenging about financial management? How did you handle the challenges? 1.19. How would they describe your organization’s relationship with the relevant USAID activity manager? Section 2: Capacity Building and Grant Management: 2.1. What do you understand by capacity building/organizational capacity development? 2.2. Please describe the capacity building model adopted by your organization and provide (as possible a diagrammatic representation 2.3. Is your organization proactive in identifying areas of capacity building support that would require external technical capacity assistance? 2.4. For those sub-partners who also provide oversight for member organizations/sub sub￾partners, to what degree do you help the sub-partner from your capacity building model so that they can apply the same techniques to their sub sub0partners as appropriate? 2.5. Is there any instances where the UGM capacity has been built by the innovations/contributions of the sub-partners? 2.6. (Refer to earlier response and first meeting) Can we further explore the understanding between your organization and USAID about the Capacity building activities to be undertaken? Please elaborate on what you have previously said and written. a) (CB Progress Reports on Partners) You have said each partner is different. Can we discuss some of the key differences? Can you direct us to the progress reports for each specific partner to support our discussion please? b) Can we discuss further the understanding between your organization and USAID about the Grant Management activities to be undertaken? Please elaborate on what you have previously said and written. c) (GM Progress Reports on Partners) You have said each partner is different. What reports shed the most light on GM progress or performance of each partner or lack thereof? 111 South Africa Umbrella Grants Management (UGM) Performance Evaluation 2.7. From the start, how clear was the anticipated number or volume of UGM sub-grantees or sub-grants for which you would provide grant management and capacity building support? (If expectations changed over time, please say when and why) (please highlight the applicable answer) VERY CLEAR CLEAR NEUTRAL UNCLEAR VERY UNCLEAR DO NOT KNOW O O O O O O 2.8. Can we discuss further the USAID/PEPFAR Monitoring and evaluation requirements for your organization, and for the sub-grantee management of the UGM? How, when and why did they change? 2.9. Communication and coordination with USAID. Let’s revisit your coordination and communication with USAID. (Refer to first response and notes from USAID discussions). Open discussion with your staff. Please take notes. 2.10.Communication and coordination with Grantees: Let’s revisit your coordination and communication with sub-grantees. (Refer to first response and notes from USAID discussions) What would you change, and why? Responses could include technological barriers, limited time, etc. Open discussion. Please take notes. THANK YOU VERY MUCH FOR YOUR TIME. IF YOU HAVE ANY QUESTIONS ABOUT THE EVALUATION PROCESS, OR WOULD LIKE TO SHARE MORE COMMENTS WITH THE TEAM CURRENTLY CONDUCTING THE UGM PERFORMANCE EVALUATION, PLEASE FEEL FREE TO COMMENT NOW, OR WRITE TO US AT THE FOLLOWING ADDRESS: UGM.EVALUATION@GMAIL.COM ALL YOUR COMMENTS AND E-MAILS WITH THE TEAM WILL BE KEPT CONFIDENTIAL AND ANONYMOUS (YOUR NAME WILL NOT APPEAR IN ANY OF THE DRAFT OR FINAL VERSIONS OF THE EVALUATION REPORT SUBMITTED TO USAID). 112 South Africa Umbrella Grants Management (UGM) Performance Evaluation GRADUATE SUB-PARTNERS Questionnaire for Graduated Sub Partners Organization: _________________________________________________________________ Date and Time: ______________________________________________________________ Names and Titles: _____________________________________________________________ Introduction Introduce self, purpose of SI evaluation. We understand that you were previously engaged in a partnership with XXX as the UGM partner through which you received funding and that now you receive funding directly from USAID. Is this correct? We would like to learn from your experiences in the UGM partnership with FHI360 to help make recommendations to USAID for an effective UGM model going forward. I’d like to ask you a few questions about your previous relationship with FHI360 as a UGM with USAID, followed by some questions about your current organizational situation. Section 1: History of relationship with UGM Partner 1.1. What does UGM or Umbrella Grants mechanism mean to you? 1.2. What was your understanding of the reasons for your organization being made a sub￾partner under a USAID UGM mechanism? 1.3. For how many years was XXX a sub partner under the UGM mechanism? (From what year to what year?) 1.4. What was your understanding of the reasons you ‘graduated’ from the UGM partnership to being directly funded by USAID? 1.5. What were the key benefits to your organization in terms of grants management while you were a sub partner under the UGM mechanism? 1.6. Do you continue to use the financial systems that were required under the UGM partnership in reporting relationships other than with USAID? (Do what degree have they been institutionalized?) 1.7. What were the main challenges in terms of grant management while you were a sub partner under the UGM mechanism? 1.8. Did your organization receive any capacity building support from your previous UGM partner? If so, how has this capacity building benefited your organization over time? 113 South Africa Umbrella Grants Management (UGM) Performance Evaluation Section 2: How Organization Has Evolved (what does a graduated organization look like?) 1.1. What are your advantages/strengths compared with organizations operating in the same programmatic area/domain as you in SA? 1.2. To what extent has your organization grown since your graduation from the UGM partnership? (Staff, coverage, delivery quality, target population)? 1.3. To what extent has your Mission evolved over time? 1.4. What are some of your well-established practices (good practices)? 1.5. How would you describe your financial strengths and challenges? 1.6. How would you describe your governance strengths and challenges? 1.7. Have you used a particular capacity building model (if applicable), e.g. to build capacity of staff, volunteers or beneficiaries? a) If so, what is the MOST common capacity building method you use? b) If so, what is the MOST effective capacity building you use? 1.8. How do you measure success? 1.9. To what extent has the percentage of USAID contribution to your overall budget changed over time? 1.10.Are other donors supporting you? Are other donors supporting other organizations similar to yours? 1.11.Looking back on your previous experience with a UGM partner, and taking into account where your organization is now, do you have any suggestions for improvements in the USAID UGM model in terms of grants management and/or capacity building to support partners? Thanks, reiteration of confidentiality. 114 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX F. CASE STUDIES Case study alphabetical and numerical list 1. AGRI-AIDS 2. ANGLICAN AIDS HEALTH TRUST (AAHT) 3. CHILD LINE MPUMALANGA 4. C… (GOVERNANCE CASE STUDY)* 5. CHILDREN IN DISTRESS (CINDI) 6. DSD EXTERNAL SUPPORT FROM PACT CONTRACTS DOH MCWH & NUTRITION 7. FOUNDATION FOR PROFESSIONAL DEVELOPMENT (FPD) 8. GREATER RAPE INTERVENTION PROJECT (GRIP) 9. HOSPICE AND PALLIATIVE CARE ASSOCIATION (HPCA) 10. HUMANA PEOPLE TO PEOPLE (HPP) 11. KHETH’IMPILO 12. NATIONAL ASSOCIATION OF CHILD CARE WORKERS (NACCW) 13. NOAH 14. SOUTH TO SOUTH 15. WORLD VISION 16. WOZ’OBONA *This partner is not identified by name for reasons of confidentiality 115 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: AGRI-AIDS Rationale for Conducting this Case Study Agri-Aids represent a rare instance of a South African non-profit program targeting farm workers with HIV/AIDS services; it is the only UGM supported partner receiving PEPFAR funds to target farm workers exclusively. The primary services are HIV Prevention and HCT. As farm workers are an under-served, at-risk population, and the agriculture sector is small but vital to the South African economy, it was deemed worthy of consideration as a case study. Quick Facts Sub-partner year of registration: 2006 UGM mechanism (+ year UGM support started): RTC (FY 2009) Organization’ s main programmatic area(s): -Direct service delivery (Prevention, Treatment, Care and Support) Direct Service Delivery – Prevention of HIV, Counseling and Testing (HCT), and Care and Support Target population: Farm workers and their families Main geographical area(s) of intervention (Provinces) Four: Mpumalanga, Limpopo, NW, FS Number of Staff: (all positions, not only UGM related) HQ Staff: 6 Full-Time Provincial: 7 Full-Time UGM percentage contribution to the organization’s budget: 100% UGM Processes and Effects: The Agri-Aids Experience UGM Application Process and Pre-Award Assessment Agri-Aids (AA) reported that it did not make a formal application in response to an APS, nor did it recall having received a comprehensive pre-award assessment by Right to Care (RTC), though it was later determined that RTC had sent a self-assessment. Neither AA nor RTC had the initial written pre-assessments of AA available, although RTC was able to provide later financial assessments. Regarding the initial award, AA recalled being invited to a meeting at USAID that led to their award, but indicated that it did not apply formally to the APS. RTC literature describes that it provides its sub-partners with assistance in four areas: 1) program management; 2) financial management; 3) grant management; and 4) M&E support. However, RTC does not present this package as a “capacity building” mechanism, per se. RTC also reports that it uses four strategies to deliver its services to partners: a) training; b) mentorship; c) site visits; and d) site assessments. AA reported there was not significant focus by RTC in areas such as governance and leadership, nor in the area of strategic planning. This seemed to be consistent with the claims made by RTC. Grant and Financial Management RTC focused capacity building primarily on grant and financial management, and M&E, including offering AA their new financial software, RightMax, in the last year. AA reported that it receives monthly visits from RTC for financial reporting. There was a steep learning curve with RightMax in the beginning. It required that AA purchase new equipment (i.et, a higher capacity printer/scanner in 116 South Africa Umbrella Grants Management (UGM) Performance Evaluation order to scan and send all receipts to RTC, and additional staff to manage the increased financial reporting burden). AA complained that the new financial software was a significant strain at first. However, AA also conceded that once it mastered the system, they found that it offered them a number of advantages compared to their previous one. For example, they can now easily document and retrieve information on all expenses online. Other grant management support from RTC was not a highly formalized or structured process. The general impression given to the evaluation team was that if AA asked for help, they got it. Starting with COP 11, RTC instituted a quarterly review process that included four steps: 1) Internal review by RTC; 2) Feedback meetings with partners; 3) Action Plan development; and 4) Implementation of Action Plan and follow up by RTC. Capacity Building As above, other than for grant and financial management, capacity building was not a highly formalized process between AA and RTC. However the AA Director said that AA received assistance from RTC as and when needed. The Director said requests were often made via phone conversations, with emails sent later to follow up. An example was M&E assistance requested from and provided by RTC. Another example cited was that RTC assisted AA to develop more detailed and “more practical” work plans that related more clearly to activities “on the ground.” AA indicated that the new work plans were a significant improvement over their previous work plans. External Technical Support Except for RTC help with selection of a vendor for M&E software development, no other specific instances of external technical support were identified. RTC helped AA select a company to design the M&E software and sort through the bids. This became the system called “Soweto Care.” With the help of RTC, AA was able to exclude the highest bid, which was approximately ten times the lowest bid. However, the software development process was not as comprehensive as it might have been. It incorporated tracking fields for HIV prevention and counseling and testing, for example, but did not include Care and Support services, for which AA received funding. As a result, it was reported that AA did not track or report Care and Support numbers adequately to RTC or USAID until last year, when this oversight was identified by the USAID Activity Manager. A seamless collaboration with RTC might have spotted the oversight earlier. Service Delivery: A major advance was that PEPFAR funding allowed AA to purchase cars, greatly increasing their mobility, which in turn allowed them to reach more farms. Due to the inherent problem of distance between farms, and the remoteness of farms, it is incumbent on AA to acquire and maintain means of mobility to expand their programs. Funding also allowed them to bring on board more staff and in turn to reach more clients. The targets increased from 4500 in 2005 to 7,000 in 2006, to 12,000 in 2012. AA added more farms as well. However, in other respects, AA might benefit from additional technical assistance. For example, AA has not yet developed its own technical guidelines, but relies instead on guidelines developed by other organizations, such as the SAG or CDC, etc. This has a possible advantage, in that AA aligns naturally with national and international HCT guidelines. On the other hand, while this is efficient, AA appears not to have developed a significant amount of internal CT expertise. 117 South Africa Umbrella Grants Management (UGM) Performance Evaluation One observation is that RTC, with its strong clinical expertise, might have engaged more with AA in terms of Quality Assurance in the field. The field visit to Mpumalanga indicated that the logistics of HCT were very good, but confidentiality measures might need to be strengthened. For example, on the farm, clients lined up very close to the tents in which initial testing was underway. Furthermore, clients who had finished initial testing were sprawled on the ground very close to the tent in which CD4 blood draws were conducted, offering little assurance of confidentiality to the clients receiving blood draws. No action was taken by AA staff or the Sister from Public Health Care until the UGM team spoke up. Defining and Measuring Success: RTC provided significant assistance to AA to help them develop their M&E system, as well as to select a vendor to develop the software. On the other hand, AA indicated that it had difficulty reporting Care and Support client numbers correctly until as recently as last year. This was due to poor data capture that relied on old M&E paper forms, in which C&S numbers were recorded under HCT fields. The USAID Activity Manager asked AA about the low C&S numbers, which led AA to look into the situation more closely. RTC apparently did not detect the problem on its own. As a quality control measure, RTC might have queried numbers reported by AA in each PEPFAR program area to ensure that the numbers corresponded to targets and funding levels. Sustainability: Agri-Aids is currently exploring options to solicit farmer co-sharing in the cost of service delivery, and seeks to develop proposals to obtain new funding sources. However, to date, it has not succeeded in attracting major new donors. On the other hand, AA is carefully aligned with the SAG. In this sense, it has a sustainable programming approach, and could conceivably obtain a higher proportion of funding from the SAG. Innovation: Agri-Aids piloted a farm-worker prevention, care and support service delivery package and has attempted to roll it out and scale it up in four provinces (Limpopo, Mpumalanga, NW, and KZN; however, AA commented that farms may be too far apart in KZN to allow this model to succeed there). Agri-Aids are also looking into “Door-to-Door” service delivery. This would involve soliciting the assistance of a NIMART trained nurse from DOH to provide ART services on the farm plot. 118 South Africa Umbrella Grants Management (UGM) Performance Evaluation Program Highlights STRENGTHS The AA program was the only PEPFAR-funded HIV/AIDS program exclusively targeting farm workers and working on farms. The program is an effective way to reach out to farmers. It first enlists their participation and support before trying to reach farm workers. Agriculture represents just 2.5 % of the South African GDP, but involves 9% of the labor force. AA has identified a key sector of the workforce that has been neglected in terms of HIV/AIDS services. The UGM, RTC, has supported AA in its outreach to farm workers. The ability to produce a significant proportion of the food needed to feed its population is vital to South Africa, as is the need to maintain employment. Investment in the health and productivity of the agriculture sector and its labor force therefore is highly advisable. AA relies on local residents as HIV testing counselors, and there is little turnover as a result. It would be desirable to continue to use local residents as counselors, though there is a need to ensure that they meet initial qualifications and also receive ongoing training and development. AREA OF IMPROVEMENT Over time, AA has developed internal systems with RTC capacity building support and USAID/PEPFAR resources. It also improved the technical quality of its programs through collaboration with CDC and DOH. There is need for greater leadership by government to address HIV/AIDS in rural areas, especially on farms and with farm workers. A further challenge is that rural residents in general and farm workers in particular, generally have less access to HIV/AIDS information and services than urban dwellers. Conversely, rural areas remain more fully under the sway of traditional healers as well as of religious sects that may propagate and maintain beliefs that are inimical to timely effective prevention, care, and treatment of HIV/AIDS. For example, some churches or faith healers claim to have the ability to prevent or cure HIV/AIDS, which may discourage uptake of HCT or treatment or the use of condoms. As such, donors like USAID, and UGMs and their partners, may need to invest in new strategies and initiatives to deliver more HIV/AIDS information and services in rural areas. The challenge is likely to require involvement and engagement of religious and traditional leaders so that, over time, they begin to support rather than oppose key HIV/AIDS messages. The National Strategic Plan for HIV, STIs and TB for 2012-2016 refers to the need for a clear strategy to address rural areas, and calls on civil society and employers to do their part to fight HIV/AIDS in these areas. However, these calls on various parties may mean little unless the government exerts leadership and ensures that a clear strategy is developed, funded, and implemented. If USAID and other donors were able to help support efforts of the Ministry of Agriculture to develop a comprehensive HIV/AIDS strategy, it might enable USAID and its partners to partner with the MOA and to intervene more effectively in this key area. LESSONS LEARNED The participation and support of farm owners/managers is crucial to the success of farm-based HIV/AIDS programs. The experience of AA is that if approached in a thoughtful and respectful manner, most farmers will participate. Future UGMs and their partners who seek to work with farmers should demonstrate an understanding of the pressures, needs, and constraints of farmers, as well as those of farm workers. For example, many or most farm workers are seasonal and/or migrant workers. They cannot afford to take off large amounts of time from work to seek health or HIV/AIDS services, as they are paid only for days or hours worked. Farm workers also need to feel that they have been consulted and have given their consent to participate as well. This would appear to be an area for potential expansion given: a) the importance of the agricultural sector to the South African economy; and b) the high prevalence of HIV/AIDS among migrant farm workers in areas. A final observation is that to the outside observer, farms, farmers, and farm workers seem to represent a setting and groups in which the rapid political and social changes underway in South Africa play out on a daily basis. One might suspect that farmers would be conservative and resistant to programs like AA. As a sign of hope that farmers and their workers can build new relationships that are more consistent with the new multi-racial future of South Africa, the UGM met a farmer who not only agreed to participate, but suggested ways to involve more farmers. For example, he recommended that AA work through either Agricultural Unions, or Farmers’ Associations, in order to present their program to more farmers, and also use these bodies to liaise with provincial governmental authorities or with the national government. Health and HIV/AIDS services, which contribute to the bottom line of farms and other businesses, might form a bridge that brings black and white South Africans closer together in areas of mutual interest. As such, investment in these services promotes the full national interest in multiple ways. 119 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM Effectiveness Summary Index (1=below average; 2: average; 3: above average) GOVERNANCE AND LEADERSHIP GRANT & FINANCIAL MANAGEMENT SERVICE DELIVERY CAPACITY SUSTAINABILITY 2.2 2.5 2.6 2.0 Documents reviewed for this Case Study: • Agri-Aids Narrative Report to RTC, Q1, 2012 • Agri-Aids COP 2011 • Agri-Aids Interviews (Pretoria and Nelspruit) • RTC COP 2011 • RTC Overview • UGM Budget and Planning, 2011 • COP 09 Initial Budgeting • Agri-Aids Org Chart • Periodic assessment(s) of Agri-Aids by RTC 120 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: ANGLICAN AIDS AND HEALTHCARE TRUST (AAHT) Rationale for Conducting this Case Study: Vana Vetu—meaning “our children”—is a faith-based OVC program of the Anglican AIDS and Healthcare Trust (AAHT). It falls under the auspices of the Anglican Church of South Africa (ACSA). The program mobilizes church communities to provide care and support for children orphaned or otherwise made vulnerable by the HIV pandemic and supports those caring for them. The program is of interest as a successful faith-based network of OVC community caregivers. Quick Facts about AAHT Sub-partner year of registration: AAHT was registered in 2003. The Vana Vetu program was rolled-out in 2007. UGM_ (+ year UGM support started): FHI 360 started supporting Vana Vetu in 2007 Organization’ s main program area(s): OVC Target population: Children and youth orphaned or vulnerable due to HIV/AIDS, their families and caregivers Main geographical area(s) of intervention: Eastern Cape, Western Cape, North West, KwaZulu Natal and Limpopo (eight dioceses) Number of Staff: 13: Six diocesan coordinators and seven AAHT head office staff; 150 volunteers UGMP % contribution to overall budget: 100% UGM Processes and Effects: The FHI 360 Experience UGM Application Process and Pre-Award Assessment The UGMP conducted a Pre-Award Assessment of AAHT to find out how AAHT worked. It looked at the organogram, systems and structures, and identified gaps as well as ways to close the gaps. AAHT used it to develop a plan to strengthen finance, human resources, and monitoring and evaluation. Subsequent assessments were conducted every year to track progress. The UGMP provided TA where there were gaps. For example, FHI360 assisted AAHT to set up a Trust and offered workshops to develop the Board. The last assessment was done at the end of 2011. Current issues include AAHT’s relationship with government, strategic planning, and sustainability plans Grant Management • Disbursements from FHI 360 to AAHT have been timely; any delays were caused by AAHT. • AAHT is responsible for compiling and submitting financial reports to FHI 360. Capacity Building (CB) • FHI 360 assisted with all areas of CB. FHI360 provided a lot of TA to the financial director, held workshops on HR systems, and ensured that policies were put in place. As a result, they hired an M&E person, financial person and HR person. 121 South Africa Umbrella Grants Management (UGM) Performance Evaluation • The assessment reviews have changed; issues are not so critical anymore. There might be a decrease of TA in one area, but another area might need attention, so AAHT hasn’t seen an overall decrease in TA; new needs emerge and need to be addressed. External Technical Support The UGMP provided various forms of TA directly related to needs/gaps/weaknesses identified in the assessment. FHI360 contracted external TA if did not have the capacity/expertise in-house. Service Delivery • AAHT dropped its OVC feeding scheme due to PEPFARPEPFAR indicator changes • The USAID Activity Manager (AM) inspired them to add new elements to the program. She said, “I’ve seen other programs doing this, why aren’t you doing this…? (e.g. child care forums)… she inspired us to do our best …” • FHI360 provided a lot of advice on programming, including resource materials, information on networks, and strengthening the program. • AAHT have shifted to a greater focus on psychological care provided by trained professionals; formerly the focus was on psychosocial care. This shift was influenced by the PEPFAR indicator changes. Vana Vetu's understanding is that such services can only be provided by a trained professional. Defining and Measuring Success • In terms of direct services to children, changes in USAID indicators meant that OVC numbers went down, but the intensity of services provided to individual children increased. • In terms of CB to deliver services, there have been many successes, due to the CB support provided by FHI 360. • AAHT didn’t have an M&E officer before; now they do. • CB of the Board in governance yielded very good results. The Board now takes ownership of its duties: “At first the board didn’t know what AAHT was about, now they are so inquisitive.” “They are all very enthusiastic; …they want to know everything.” • Before, the DOVC didn’t know much about computers, but now they have their own laptops and can generate reports. They can also budget and manage the funds allocated to their Diocese. They are responsible for a budget. They have been raised to a professional level. The program has positively affected all staff on a personal level. Staff have confronted their own childhood hurts; through training, they have been able to release it. Sustainability • AAHT attended FHI 360 sustainability workshops held at national and provincial levels. At district level, the DOVC attended. They need to build capacity to fund-raise. They have accessed Global Fund money, but said this was not a direct result of the sustainability plans. • Each parish should have a fundraising plan and a strategy to implement it. This requires CB in fundraising and bookkeeping to ensure that donations are well managed. Some parishes have been registered as non-profit organizations (NPOs) to fundraise. The basket of services provided to OVC aligns with national policies, and sector best practices on OVC care. AAHT have consulted with the Children’s Institute to ensure that their policies and services are aligned with the Children’s Act. Innovation • NA. 122 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights STRENGTHS • FHI360 has been very supportive of AAHT: “a pillar of strength”. When AAHT have quarterly review meetings with USAID, AAHT says: “In the reviews they are … on our side…!” “Our success is their success.” • Support is given to AAHT as a whole, not just to Vana Vetu. AAHT has benefitted; their systems have been strengthened. AREAS OF IMPROVEMENT • AAHT would have liked more help building relationships with government. • The provincial PEPFAR liaison (PPL) has been helpful by bringing together the partners and government. FHI360 mainly brings in Social Development; it would be good to include Education, Health, and Home Affairs. LESSONS LEARNED • The first year as sub-partner, they had to get the language of PEPFAR and understand the systems. It was also AED first year as UGMP; a “as we grew together we understood more clearly each other’s role” • It is important to be open and transparent in the pre-award assessment and subsequent assessments. AAHT adopted this approach. UGM Effectiveness Summary Index (0=below average; 1: average; 2: above average) Governance and Leadership Financial and Grant Management Service Delivery Sustainability 2.0 2.6 2.2 2.0 Sources of information: • AAHT Mid-term evaluation. Prepared by Greater Capital 24 March 2011. • AED UGM mid-term evaluation May - July 2010 • FHI 360 UGM Annual report October 1, 2010 – 30 September, 2011 • interview with Vana Vetu program staff, 2 April 2012 • http://www.anglicanaids.net/vanavetu.htm 123 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: CHILD LINE MPUMALANGA (CLMPU) Rationale for Conducting this Case Study: Child Line Mpumalanga (CLMPU) was founded to address a specific gap in child protection services in the Mpumalanga province. The organization has grown rapidly in a relatively short period of time. Looking at this UGM Sub-Partner’s experience is likely to provide a sense of what it takes for a relatively small and young organization to expand rapidly within a short period of time and what challenges are associated with such UGM-supported growth. Quick Facts about Child Line Mpumalanga Sub-partner year of registration: 2000 UGMP (+ year UGM support started): Pact since 2007 Organization’s Main program area(s): OVC Partners: DSD, SASSA and Home Affairs Primary Target population: OVC (including their families and caregivers) Main geographical area(s) of intervention: Mpumalanga Number of Staff: 148 now, up from 3 before the UGM (this includes the child care workers and volunteers) UGM % contribution to the overall budget: Approx., US $4 million (as of October 2011); Previously was $1.03 million (2008-2010); UGM Processes and Effects: Child Line Mpumalanga Experience • Each province in South Africa (except Mpumalanga) had a child crisis line. Child Line Mpumalanga (CLMPU) was created in 2004 to fill this gap. • Initially, the organization was called Life Line and had only three staff. The organization soon rapidly grew from one social worker to four supervisors, each with several social workers. • Child Line is not allowed to provide economic support using governmental funds to non￾South African children. One solution was the creation of food gardens; • Another challenge is the mobility of all the needy (often undocumented) children it aims to serve. CLMPU have put in place a mobile clinic mechanism; • What is new about the CLPU program is having translated an emergency model of psycho￾social care into a broader community development intervention approach. UMG Application Process and Pre-Award Assessment • CLMPU started receiving UGM funding in 2007 to implement OVC support activities and has been one of Pact’s partners since then. 124 South Africa Umbrella Grants Management (UGM) Performance Evaluation • After undergoing a Pre-Award Assessment conducted by Pact at the start of the UGM partnership, CLMPU revised its Human Resources and Financial Management Policies; • Subsequent assessments were conducted each year. Overall, Pact field visits were considered useful. They pushed CLMPU staff to integrate all their programs. • The relationship with Pact has also been quite good • The relationship with the USAID Activity Manager (AM) has been quite good and very open. Governance and Leadership • The Board of Directors meets several times a year. Board member positions are time-bound (average duration: less than 2 years). • Specific guidelines on decision-making exist and everybody is aware of them; • All the staff members have job descriptions detailing Key Performance Areas included in their TORs. Performance reviews are conducted every 6 months Financial and Grants Management • Child Line’s progress in financial and grant management and reporting is reflected in the evaluation of its spending rate. • The disbursement of UGM funds has been timely. After Child Line submits its financial report, it should take 10 days to receive funds. In reality, it takes about 14 business days. Funds are often in right before they need to pay salaries to staff. • It is still not clear at times what an allowed expense is. As a result of stringent guidelines, financial management has improved. They use USAID standards with all other donors. • Sound financial management is so ingrained in their day-to-day practice that they look at it as their own practice and not a specific tool imposed on them by a donor. • In general, it took Child Line 3-4 weeks from the notification of grants disbursement-related non-compliance instances (e.g., reporting) to address them. • One of the issues not settled yet relates to salary sharing for individual positions; • Disbursements were consistent with their implementation plans most of the time (90%). • CLMPU never experienced delay in submitting financial reports to Pact. They employed an extra person to capture data and the funding allowed them to do that. • CLMPU was given a very large funding upscale in October 2011. That allowed CLMPU to procure vehicles. They were grateful but their absorption capacity was seriously challenged. Capacity Building (CB) • CB often happened informally. Pact staff would give CLMPU their feedback back on documents. They would have an intense exchange on this for two weeks. • Pact was very responsive and involved, and did not approve strategic documents until they were satisfied. • Child Line definitely learned USAID language (“youth” for example in South Africa is up to 35 – while in the US it is less than 18). • They were able to transfer learning from this annual work plan and budget development process to the management of any other funding they receive. 125 South Africa Umbrella Grants Management (UGM) Performance Evaluation • The most effective method was Mentoring through Site Visits (one full day) Child Line has also been working hard to educate the public at different levels (provinces, districts, chiefs, counselors and communities) on what a social worker does. Service Delivery • The UGM partnership helped them meet or exceed their OVC service delivery objectives. CLMPU started offering services, such as leadership groups (10-18) and memory books; • Although not targeting new population groups, CLMPU staff have increased the number of the children they serve (from 6,800 to 8,000) and included two new geographic areas; • Implementation of activities was never interrupted. In case of minor delays in funding disbursements by Pact, CLMPU occasionally tapped into their own resources. Defining and Measuring Success Indicators used to measure capacity building/development at Child Line Mpumalanga include: • Number of trainees/referrals/how crèche are run (registration) • Pre-test and post-test (in case of trainings) • Beneficiaries visits based on a random selection of children/households twice (before and after the intervention). • Retention rate of social workers and volunteers • Number of calls to the Child Line (proxy indicator for increased awareness) • CLMPU staff also go to sites with care workers and observe how they operate • Every six months, the director goes to each site and meets with the local leaders; • Of CB support provided by the UGM, data quality was regarded at the most useful Sustainability • The organization has not secured new funding due to UGM resource mobilization support; • The province has not signed an MoU; they would then need to support CLMPU • Some costs are shared by other, including: salaries of director and social workers • Computer services are provided by a local company; all venues for trainings are donated by the Travel Authority • UGM funds percentage contribution to the CLMPU budget: 50-70% Innovation • Significant innovations that have emerged from the CB support provided by the UGM are: • M&E providing data to all sections • Standardized reporting Suggestions for Enhancing UGM Responsiveness and Agility 126 South Africa Umbrella Grants Management (UGM) Performance Evaluation • A good strategy would be to give sub-partners a better-defined plan and say how visits will be structured. Pact is reactive/responsive. Better planning should free up sufficient time. • Yearly meetings with USAID staff would be very beneficial • More and earlier orientation would have been helpful. This should not take place a year after implementation has started. Highlights Key Challenges: • Their absorption capacity was challenged by rapid funding increase and delayed disbursement • Financial reporting is a challenge for staff whose salaries are partially covered by other sources Strengths: • As a small sub-partner, it was able to grow fast within a short period of time Area for Improvement: • Oversight of hiring procedures Lessons Learned and Promising Practices • Government’s cost sharing for all social workers employed by the organization • Clearly articulated HR policy, spelling out procedure to mitigate conflict of interest • Engagement with government: Very Good Challenges with government engagement: • The Provincial Government might be hesitant to sign a Memorandum of Understanding with CLMPU to prevent itself from having to take on CLMPU expenses in the future How to align with government plans and priorities: • Sign a MoU with DSD UGM Effectiveness Summary Index (1=below average; 2: average; 3: above average) GOVERNANCE AND LEADERSHIP FINANCIAL AND GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.8 2.5 2.4 2.6 127 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: C* (GOVERNANCE AND LEADERSHIP CASE STUDY) The evaluation team considered how to review and explore the issue of problems at the governance (Board) and leadership (CEO or Executive Director) levels. Problems at this level are highly sensitive and their open discussion in a public document does not bode well for the quick resolution of any pending conflicts. Therefore, this team presents a situational analysis of common problems, and discusses possible ways forward for UGMs, but does not specifically identify any actual partner or UGM. Commonly encountered Board level conflicts are of three general types: 1. Conflict between Board of Directors and an Executive Director 2. Conflict among or between Board members including the Board Chair 3. Conflict between Staff and Executive brought to the Board of Directors Rationale for Conducting this Overview of Governance and Leadership The rationale for the overview is that Governance and Leadership are critical to the success of NGOs. Conversely, failures or serious conflicts “at the top” can rapidly destabilize or cripple an NGO, and demoralize its staff. USAID and any future UGM may have to consider how to handle potential board level issues if any crises arise. UGM have made use of “prevention” efforts such as board training and conflict resolution, but there have been crises even in organizations which benefited from such CB efforts; additional steps are needed. The donor (USAID) and the UGM (any) may not be party to the conflict or problem, but their resources and reputations may be at risk. Also, such crises or conflicts may result in brief or longer￾term interruption or cessation of operations, which may deny services to those in need, and also result in failure of organizations to achieve targets or results for which they, USAID, and the UGM are all accountable. Therefore, it is of inherent interest to USAID and the UGM to look at governance and leadership closely and regularly throughout the life of project. USAID and any UGM need the tools, processes, and capability to undertake this task with any partners. Situational Analyses – Governance and Leadership Challenges In one case, a UGM worked with a national non-profit partner that had come to be seen as one of the UGM success stories for the quality and quantity of services it provided. This national organization, which has provincial affiliates, is a large non-profit provider of child protection and child care services. Starting in 2005-2006, it had experienced some difficulties with its financial management, and had lost some of its donor support. Needed staff changes were delayed, and it became clear that new leadership was needed. *This SP is not identified by name for reasons of confidentiality 128 South Africa Umbrella Grants Management (UGM) Performance Evaluation Then, with USAID/UGM support, and the recruitment of a new Executive Director (ED) in 2007, it finally resolved past problems and appeared to be set to move to another level of service provision and capacity. From 2007 until 2010, it remained on track and performed well both in terms of services and internal management. The ED developed a close and collegial relationship with the Chairman of the Board of Directors (BOD), who took an active interest in the operations of the non-profit. This had the potential to provide greater BOD support to the ED, but also carried a risk. The classic risk is with over-involvement of the BOD in operations (“the camel’s nose under the tent”), as opposed to a BOD focused on strategy, policy, and vision. Beginning in 2010, there were early signs that the non-profit was again having some difficulties, including delays in financial reporting. Then there were challenges at the governance and leadership level. In 2011, open conflict occurred between the Board and the ED. The Board Chair, seen as the strongest member of the BOD, took action. As often happens, this open conflict resulted in the departure of the ED. Soon after, the UGM temporarily suspended further PEPFAR funding to the non-profit, resulting from concerns about the non-profit’s financial management and oversight. The non-profit and its local affiliates continued to provide other services not funded by USAID. The situation demonstrates that success and capacity building do not preclude the possibility of serious difficulties in terms of governance and leadership. It further demonstrates that these areas are critical for capacity building. They bear very close watching due to the risk that conflict at the top will destabilize or even to paralyze an organization. The UGM was never cited as having any role in or responsibility for the conflict or its resolution. The UGM has indicated that its capacity building model is driven by requests from its partners. This approach has both advantages and disadvantages. On the plus side, it ensures that its services are welcomed and valued. On the minus side, it may result in a situation in which non-profits ignore or brush aside serious underlying needs or concerns that actually need to be addressed. This could be because non-profits are unaware that they have a problem or need, or are aware, but the Board and/or the Leaders are unable or unwilling to deal with the problem. 129 South Africa Umbrella Grants Management (UGM) Performance Evaluation Lessons Learned GOVERNANCE - BOARDS AND EXECUTIVES • Good governance and leadership are absolutely critical to the success of any organization, for profit or not. Even if an organization has long enjoyed good governance and leadership, it is not immune to the possibility of conflict at the top. A clear vision and mission, well-designed programs, and the dedication of senior and middle management and field staff may be enough to see an organization through a crisis of governance and leadership, but it is best to avoid the crisis through good planning and via board development. Good grant and financial management as well as service delivery capacity may also help to sustain an organization during a difficult transition. “Outside parties” even if interested, may be extraneous to or excluded from the resolution of internal board conflict, regardless of their degree of involvement or investment. In this instance, USAID and the UGM were “sidelined” and were not directly “party to the conflict.” This kept them out of the conflict, which had advantages, but it meant they could not effectively intervene to resolve it. • A more balanced, well-trained board, composed of many members who enjoy comparable levels of influence and authority, may be preferable to a board in which one or a few members tend to dominate board discourse, either through their seniority or their level of influence over the others. Also, clear and consistent BOD boundaries, written guidelines, as well as clear definition of roles and responsibilities of board members and executives, are highly advisable. • Guidelines suggest that conflict involving the BOD must be identified as early as possible, when the conflict is small and containable, and addressed immediately. Generally, once the conflict is a major one, and comes out in the open, the risk of serious damage or disruption is significant. Once conflict between boards and executives arises, the following options may be considered: • Mediation between the board and executives – preferably internally, only then externally • Deal with conflict confidentially, but not in secret, and be transparent about the outcome • Guided facilitation of a conflict resolution process if both parties are amenable to it ISSUES OR APPROACHES THAT USAID AND/OR UGMS MIGHT WANT TO CONSIDER OR REQUIRE IN THE FUTURE • Board assessment and board development might be made compulsory, with red flags identified; require resolution of conflicts as a condition of initial/ongoing USG funding • However, the literature suggests that ideally, Board development and governance must be “owned” by the board and NGO it governs, not seen as controlled by outside influences • UGMPs should encourage UGMSPs to develop boards, ensure leadership and boards understand best practices, and teach leaders and boards to recognize potential problems. • USAID and UGMPs should continue to exercise “due diligence” in board and leadership matters, i.e., put in place adequate safeguards, but not “police” NGO boards or leaders. • A case in point for the potential risk of a donor such as USAID or a UGM “over-stepping” its boundaries in trying to influence or “police” NGO boards and leaders is suggested by an article about South African government concerns over undue outside influence on NGOs. RECOMMENDATIONS • Situations to avoid include a BOD dominated by one or a few members, or a BOD selected and or guided inappropriately by an executive • Review of board guidelines and term limits might also be made compulsory in future, with annual training or review or roles/responsibilities needed for ongoing BOD membership • Board guidelines might spell out conflict resolution procedures, and members of BODs might be asked to sign a code of conduct indicating they will agree to conflict resolution • Apply guidelines and requirements to UGMSP Leaders in their dealings with BODs. UGMPs could monitor the boundaries between Leadership (Operations) and BOD (Strategy) Broader board development efforts by USAID and the UGM might be considered. For example, rather than focusing on individual partners and their boards, the UGM might develop links, or set up a web application that seeks to do one or more of the following: • Link to or support existing mechanisms to support board and leadership development. Possible starting points include NGOPulse/Sangonet • Find or create online board member recruitment applications to solicit new board members; link them to training and mentoring relationships with experienced persons 130 South Africa Umbrella Grants Management (UGM) Performance Evaluation Resources consulted regarding Board of Directors • http://www.ngopulse.org/sites/default/files/draft_code_for_npos-d3.pdf Draft code of conduct for South African NPO Boards, August, 2010 • http://www.fsg.org/tabid/191/ArticleId/76/Default.aspx?srpush=true Advancing Good Governance: How Grant-makers invest in the Governance of Nonprofit Organizations • http://www.ngopulse.org/press-release/good-governance-learning-network-ngos-not-puppets￾donors NGO’s not Puppets (article from 2005 criticizing outside NGO influence) • http://www.ngoconnect.net/documents/592341/749044/NGOTips+- +Fostering+Effective+NGO+Governance Fostering effective NGO Governance • http://www.wtrc-tmed.org/resources/Board%20Games.pdf “Board Games” • http://www.tgci.com/magazine/resolving%20board%20conflicts.pdf Board conflicts • http://collegeofcontinuinged.dal.ca/Files/NP_ManagingConflict-AGuideforVolunteerBoards.pdf A non-profit guide to managing board conflict • http://www.fieldstonealliance.org/client/tools_you_can_use/12-07-05_board_conflict.cfm Fieldstone alliance guide • http://www.cof.org/files/Documents/Governing_Boards/Board%20Briefs/CEOsOnBoards.pdf Putting CEOs on the board? • http://www.governancematters.org/data/org/56/media/doc/4695_gm_bl_board-edtensions.pdf governance matters.org Governance matters 131 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: CHILDREN IN DISTRESS NETWORK (CINDI) Rationale for Conducting this Case Study CINDI (Children in Distress Network) was founded in 1996 and is a network of over 300 member organizations working with OVC. Within CINDI, The Pact UGM funds the May’khethele Project, a consortium of organizations working in high schools in greater uMgungundlovu District to support OVC. Quick Facts about May’khethele Project Sub-partner year of registration: 1996 UGMP (+ year UGM support started): Pact, since 2007 Organization’ s Main program area(s): Capacity Building Direct service delivery: Orphans and Vulnerable Children (OVC) Partners: Community Care Project, Lifeline and Youth for Christ YFC-KZN). Sinani exited as a partner September 2011. Target population: OVC who are in or out of school, their families or caregivers, others Main geographical area(s): uMgungundlovu District in KZN Number of Staff funded by UGM: 3 (OVC Program Manager, M&E Officer, Bookkeeper/Administrator) UGM % contribution to overall budget: PEPFAR funds are nearly 100% Pact commented that this is one of few consortia that held together, consistently exceed targets, and performed well. The Joint Management Team monthly meetings are a key to its success. May’khethele program areas include school based HIV prevention education, linkage to HCT, referrals and linkage to voluntary Medical Male Circumcision, CD4 tests, and referrals to ART. Support of OVC includes: school based support groups and homework clubs; referrals for general health care; school fee exemptions; in-kind items such as uniforms and school supplies; after school tuition clubs and legal assistance for child protection; home visits; and household economic strengthening as well as help with food security through referrals. UGM Processes and Effects: The CINDI May’khethele Project Experience UGM Application Process and Pre-Award Assessment The May’khethele Project was funded under Pact UGM from 10/2007 but did not receive funding until 01/2008. USAID delayed disbursement due to the perceived unsuitability of the program manager that was initially hired. 132 South Africa Umbrella Grants Management (UGM) Performance Evaluation In February 2008, Pact conducted a pre-award Organizational Capacity Assessment, with May’khethele Project staff and the consortium NGO partners. The assessment identified M&E challenges and helped new staff understand key areas in which they would receive technical support from Pact. The project now uses standardized tools for data collection and has an MER plan. Financial and Grants Management The current PEPFAR funding allocation is $800,000. Fund disbursement is in timely monthly tranches, other than the first quarter, when no money was disbursed until May’khethele replaced the program manager. The first year, the project struggled with unsupported costs. Pact assisted them to problem-solve documentation issues related to field work. Solutions were found; the project views this as an indicator of increased financial management capacity. Capacity Building (CB) CB has been received in Finance, MERL, programs, governance, and organizational development. Pact makes annual site visits. The members of the Joint Management Team participate. Pact has provided a useful template to develop implementation plans. The project sets collective targets. The JMT divides funds across organizations based on targets and attempts to balance resources for different types of services. External Technical Assistance The project did not receive external TA, and was unaware of this option. Service Delivery In FY 2010-2011, May’khethele Project surpassed targets, despite halting of onsite VCT at schools and Sinani’s early close out. Targets have been exceeded every year. One challenge has been differences in the PEPFAR fiscal year and school calendar. PEPFAR’s first quarter is the busiest school term, when students prepare for exams; the project decided to train field workers when students are less accessible. Defining and Measuring Success UGMP funded training of field workers on data quality management and use of standardized data collection tool improved data quality. The data flow from the field to the head office has improved. At least 20% of data is verified by the CINDI M&E officer, and a PEPFAR M&E fellow, with a further 5-10% verified by partners. The Project tracks eight key PEPFAR indicators and non PEPFAR indicators. Sustainability Some consortium partners have secured funding from DSD. DOH absorbed 19 rape crisis counselors trained by the project and stationed at District hospitals. Two project facilitators were hired by the schools where they were stationed. 133 South Africa Umbrella Grants Management (UGM) Performance Evaluation Sustainability has been mostly enhanced through partnerships: • School administrators linked with government departments and NGOs • LIMA and the National Development Agency (NDA) provide agricultural support to schools and homestead gardens for training to develop food gardens. Innovation The Project added male medical circumcision (MMC) education and offers transportation to MMC for male high school students in 2010. Highlights KEY CHALLENGES • A new requirement for HIV prevention education sessions to be provided in groups of less than 20 learners led to a need to revise the original service delivery model. • In August 2011, the DOE prohibited HCT in primary school environments. This project had to change its approach. • The DOE’s ban led three of four consortium partners to shift to high schools; the last partner preferred not to make this shift and withdrew from the collective. STRENGTHS • JMT members learn from each other’s experience and are accountable to each other. • There are outstanding relationships with DOE, DOH, DHA, and DSD AREA FOR IMPROVEMENT • Sustainability; May’khethele hasn’t found new funding yet LESSONS LEARNED AND PROMISING PRACTICES • Promoting MCC in schools created a testing advantage for boys, and an unintended gender imbalance. May’khethele now encourages all boys and girls to go for VCT. • May’khethele learned to train people and then mentor them; changes take time. • Have student peer leaders identify OVC. There are peer leaders for grades 8-10, and OVC support meetings every week, as well as a trained a teacher in every school. 5. UGM Effectiveness Summary Index *During piloting of the index, scores for some questions were determined to be NA GOVERNANCE & LEADERSHIP * FINANCIAL & GRANT MANAGEMENT* PROGRAM SERVICES* SUSTAINABILITY 3 2.67 2.4 2 134 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: DSD: UGM CASE STUDY: EXTERNAL TECHNICAL SUPPORT THROUGH UGM Rationale for this case study While two UGM partners (Pact and FHI 360) provide in some instances external technical support to SP, Pact is unique among current UGM mechanisms in its provision of external technical support for two government departments: Departments of Social Development (DSD) and Health (DOH). This case study looks at lessons learnt and recommendations from two consultants who provide a range of external technical support to DSD, DOH, and SP through UGM mechanisms. UGM Processes and Effects: External Technical Support Through UGM Experiences of External Technical Support Consultants Two external technical support consultants were interviewed as part of the UGM evaluation. Consultant #1 one has provided direct support to DSD and an NGO supported by Pact. Consultant #2 has provided support to DOH at national, provincial and district levels through Pact and has worked to a lesser extent with Right to Care. Consultant #1 has been contracted by Pact on behalf of DSD to develop a proposal for the roll-out of a new cadre of social welfare workers (the proposal resulted in Treasury funding to train 7,500 training new youth and child care workers; USAID through Pact separately organized an expert to cost the proposal) and by Pact on behalf of a Pact SP to conduct an evaluation of and develop standards and indicators for a program model. Consultant #1’s experience was that the hire for the DSD assignment was faster than the assignment for the SP; this may have been due to an urgent deadline for the DSD proposal development. For both the Pact contracts, the consultant was specifically asked for by DSD and the SP; to her knowledge, there was no open bid. In both of these consultancies, there was no element of capacity building for DSD or NGO staff; she believes she was hired to complete tasks that the Department or NGO lacked have time and resources to do directly. She sees trade-offs in terms of benefits to an organization to have all needed expertise in house in the form of permanent staff or to be able at times to rely on scopes of work that are outsourced to an external, short term consultant. Consultant #2 has worked with two UGMs and observes that “Pact is quick, easy, flexible and not a micro manager.” From his perspective Pact has been financially meticulous, otherwise supportive and flexible. Both consultants have the experience that some consultancies are sole sourced and others are tendered; the rationale behind different strategies for different TORs is unclear. However, most contracts do contain clear TORs and deliverables. Consultants had different experiences in terms of who among USAID, Pact, and DSD or DOH or NGO reviewed, approved and signed off on their work. Consultant # 2 has worked with Pact for 3.5 years; an initial 6 month contract was amended and extended. He was asked by NDOH to provide support to the 18 priority, underperforming districts, poorest districts in country and his eventual focus has been on two synergistic projects in eight Eastern Cape districts, where he has developed longstanding relationships. These projects took time to form, and Consultant #2 has appreciated Pact’s flexibility during needs clarification, design and 135 South Africa Umbrella Grants Management (UGM) Performance Evaluation implementation. His experience is that the second UGM partner he worked with tended toward micro managing with unhelpful rules. Health Systems Strengthening Support to Underperforming Districts Two innovative projects designed and implemented by an external technical support consultant, based on needs identified by health providers and administrators working in rural districts, are briefly described here as examples of systems strengthening provided by an external consultant. The Rural Doctors Support Program, had start-up support from Pact and implementation support from the Clinton Health Access Initiative (CHAI), and then was taken up by the Foundation for Professional Development (FPD). The Effective Management and Leadership in Resource Poor Settings Program focuses on empowerment and support for 80 district managers working in rural peripheries and consists of four 2-day workshops with roughly six weeks in between to allow managers to apply new learning before sharing experience and moving to the next issue. After current PEPFAR support ends 09/2012, this program will continue and scale up in 16 districts under the Health Systems Trust. With both programs, USAID funding through Pact has given rise to ongoing programs that will be continued by other partners. Recommendation from this external technical support experience • Future UGM explore ways to integrate the proven practice of structured workshops with strategically planned time learning between sessions into other capacity building TORs where applicable. Building the professional capacity of doctors and providing both personal and professional support for managers, and then bringing them together for improved communication represents detailed sequencing with pedagogic purpose and represents a longer term capacity development strategy for the country. Recommendations for Consultant Selection under New UGM Mechanism • UGM should strive for balance between seasoned consultants who are already familiar with the working context (relationships, politics, and context) and fairness and openness in a more competitive selection process. This may involve working with DSD to open up bidding to additional new consultants, perhaps beginning with small contracts to allow them to build up experience, and then allowing them to progress to bid on larger ones. • Some TORs may be better suited for sole sourcing, whereas others are better framed to be competitive. A UGM needs to be sufficiently flexible to weigh the benefits of both approaches, listen to the experience of government colleagues, and consider timeliness and urgency of the need. Timeliness might be enhanced if the UGM were to allow Departments to put forward any consultant preferences, with a rapid vetting process to follow. This accelerated process could be considered for scopes of work within a limited funding level. • UGM communicate to outsourced external support consultants that USAID is the donor. Often external consultants view Pact as the donor and are unaware of the UGM mechanism. • All contracts should have well developed TORs; as much as possible, Departments or NGOs requesting external support should be directly involved in any improvements made to the original TORs, to build their capacity. • Depending on the cost and complexity, UGM could consider developing an external TA database that includes consultant ratings based on past performances, with a designation for preferred consultants who have a strong performance history with DSD and/or DOH. • UGM actively seek seasoned, experienced professionals, e.g. retired social workers or retired DOH managers, who could provide skilled mentoring as part of their range of technical support. 136 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: FOUNDATION FOR PROFESSIONAL DEVELOPMENT (FPD) Rationale for Conducting this Case Study: Established in 1997 by the South African Medical Association and registered as a separate legal entity in 2000, the Foundation for Professional Development (FPD) graduated from the UGM program in 2008. FPD’s transition from a USAID sub to a prime partner was a relatively uncomplicated process; FPD was able to benefit from a shift in status thanks to the strengthening of its financial and service delivery capacity during the three years of UGM support. Learning more about the history and dynamics of this indigenous organization is likely to yield valuable lessons on what makes a sub￾partner a solid and financially sustainable organization. Quick Facts about FPD Sub-partner year of registration: 2000 UGM mechanism supporting the organization: Pact since 2005 Organization’s Main programmatic area(s): - Strategic Information (SI) - ART: $9,015,000 (2006-2007) -Medical Transmission: Blood Safety; Medical Injection Safety; - PMTCT (non-UGM funded) Capacity Building -Provider of government accredited courses in a variety of domains (115 courses) -30,000 of students (the majority being females and previously disadvantaged individuals) within and outside South Africa Target population: PEPFAR funding is for ART and PMTCT clients and others above; Main geographical area: NA from 2010 COP; 2011 COP not yet published Number of Staff: Over 1,000; in 2010, the number of FPD staff started decreasing Budget as a prime: US$ 155 million ($27.2 million in COP 2010) 137 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM Processes and Effects: FPD Experience Governance and Leadership • FPD is a well-structured organization that benefited from the UGM pre-award assessment conducted by Pact; Four clusters of functional departments report to the Managing Director and have enhanced management of its numerous operations; • The Managing Director has the overall responsibility for managing FPD in accordance with the strategic direction provided by the FPD Board of Directors; The Managing Director also acts as Head of the Special Projects Cluster; Financial and Grants Management • As UGM disbursements were consistently timely, FPD was able to achieve two primary objectives: first, the rapid escalation of ARV treatment for individuals newly diagnosed with HIV; and second, the development of strong collaborative partnerships with a number of indigenous organizations across the country • The financial and grants management capacity developed by FPD under the UGM reached such a satisfactory level that, once it graduated from the UGM program and received direct USAID funding, it eventually received nearly $155 million (the second largest grant ever given by a donor to a South African indigenous organization) • FPD feels capable to become a UGM itself; unlike the current funding modalities, though, were FPD to administer grants to HIV and AIDS organizations in South Africa, it would require grantees to match the received funding with an equal amount derived from independent (non-USAID) sources Defining and Measuring Success • FPD staff already possessed good management skills in the late 1990’s • However, as most of its funding came from the South African Government funding before becoming a USAID prime, FPD had never emphasized the role of monitoring and evaluation as much as it was required to do by the UGM mechanism in 2007 • As a result of the greater scrutiny under PEPFAR (in terms of financial accountability and service delivery effectiveness), FPD says it remarkably improved its performance Capacity Building Model • FPD staff perceived the UGM as having a clear developmental purpose unlike other donors; By contrast, European Union financial reporting requirements are more relaxed; selection of beneficiaries (as with the Global Fund) is pre-determined by a government-sponsored coalition • Likewise, through UGM support, trainees learned about Strategic Information to help operationalize their plans, through the use of action research methodology (each student needed to implement a real-life project) • FPD has incorporated some of the best available international practices in capacity development in its numerous training curricula; It has made them available locally to a much larger audience: 138 South Africa Umbrella Grants Management (UGM) Performance Evaluation - Before designing training course for district health officers, for instance, FPD conducts a learners’ needs assessment, then, through its collaboration with Pact, it helps clients develop an operational plan and a M&E framework - FPD staff organizes monthly meetings with the district to reassess trainees’ capacity and, through regular interactions, it strives to enhance trainees’ project management capacity further • FPD has found middle-junior level trainees see this as an opportunity; recently, older ones began to attend and express genuine interest in capacity building; • FPD has started to review some of its management development programs and simplified for managers at lower levels; FPD Technical Assistance focuses on decentralized entities • FPD values the role of district health personnel and plans to provide an increasing amount of technical assistance to them • Health care is constitutionally a provincial responsibility; provinces almost have 90% of resources and allocate them to districts. However, budgets are allocated on historical principles, not zero-based budgets; FPD seeks to promote zero-based budgeting • To fill health care gaps, FPD has been working primarily in six districts, developing strategic plans and annual operations plan • FPD is trying to foster a culture of health management; in past, doctors and nurses were assigned posts with no management training; FPD is enhancing the capacity in this area Cost-sharing Cost-sharing modalities vary; FPD offers scholarship to students but does not pay their accommodation, travel or per diem; On the other hand, districts pay for all training costs; FPD clinical and management Mentors who visit hospitals must get government support at some point; The new graduate status and the ongoing partnership with Pact The UGM graduate status attained by FPD has not altered the quality of the relationship established with Pact; For instance, the Collaborative agreement signed with Pact to provide M&E technical assistance translated into joint delivery of training courses; Pact handles M&E modules and, based on a revenue-sharing model, FPD handles the modules more directly associated with service delivery; Cost-effectiveness • FPD says its courses cost less than those offered by most other institutions • The effects of FPD training appear to be good, giving FPD a comparative advantage; While only 21% of the 7000 nurses working for the Department of Health (DoH) were able to initiate ART after non-FPD courses, 69 % of FPD trainees (some 500 students) were able to initiate ART right after training; • By targeting in large numbers and by contracting well-respected experts on a short-term basis, FPD recruited the top professionals while keeping costs down; 139 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights KEY CHALLENGES • Provincial and District low strategic management and budgeting capacity; local health budgets are allocated based on historical trends, not rationally and evidence-based as they should be STRENGTHS • Capillary network of trainings and trainers across the country (both urban and rural) • Wide network of partners (both institutional and private) AREA OF IMPROVEMENT • Expanding capacity building programs in new districts LESSONS LEARNED AND PROMISING PRACTICES • Cost-sharing with Pact; Establishing partnerships outside of South Africa ENGAGEMENT WITH GOVERNMENT • Very good; Coordination with provincial Departments of Health (DOH) takes place through Memorandums of Understanding (MOUs) CHALLENGES WITH GOVERNMENT ENGAGEMENT • Health management not widely appreciated and understood as it should; HOW ALIGNED WITH GOVERNMENT PLANS AND PRIORITIES: • Very well aligned as demonstrated by the large number of nationally accredited courses and the high request for trainings from DOH and other national department’s personnel UGM Effectiveness Summary Index GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 3 3 2 3 140 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: GREATER RAPE INTERVENTION PROJECT (GRIP) Rationale for Conducting this Case Study This small organization has supports rape survivors, one of the most vulnerable population groups at risk of HIV infection in South Africa. Learning more about GRIP’s experience with the UGMP is likely to provide some interesting information on how to use this grant management mechanism as a vehicle of programmatic innovation and expansion as well as an organizational development for relatively smaller non-governmental organizations. Quick Facts about GRIP Sub-partner year of registration: 2000 UGMP (+ year UGM support started): AED since 2007; FHI 360 since 2011 Organization’s Main program area(s): MARP Partners: DSD Target population: MARP (Rape Survivors and CSW) Main geographical area(s) of intervention: Mpumalanga Number of Staff: (7 staff and over 100 volunteers) UGM % contribution overall budget: more than 70% UGM Process and Effects: the GRIP Experience Understanding of UGM • UGM is a demanding mechanism not always considering the needs of staff working on operations; • Some of the in-person meetings held with the UGMP staff and a number of consultants these past few years could have been easily arranged over the phone, thus not taking time from field work; • Deadlines for the submission of deliverables or ad hoc information are very stringent UMG Application Process and Pre-Award Assessment • GRIP became a UGM sub-partner in 2007 and have been a partner with the FHI 360 ever since; • GRIP underwent a Pre-Award Assessment from FHI 360 at the start of their UGM partnership; 141 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Overall, their relationship with FHI 360 has been quite good. However, the quality of the relationship had changes quite a bit, allegedly because of the upcoming UGM close-out • The relationship with the USAID Activity Manager was good. Governance and Leadership • The Board meets more than once a year. Board members positions are not time-bound (some have been there since 2000). Good attendance is a criterion for staying. There are processes in place to mitigate conflicts. All staff have specific written guidelines available to them for this purpose; • No Mission statement in the constitution – GRIP is also a trust (that might be one of the reasons); • Decision-making within GRIP is transparent. The process is informal: everybody knows there is a set time or date (Friday meeting) to discuss matters, especially in relation to their Action Plan. • All their staff job descriptions include detailed Key Performance Areas. • Consultants recruited by FHI 360 were supposed to help out with the strengthening of GRIP HR policy (including salary benchmarking) but the work was never completed. Financial Management and Grant Management • Disbursement of UGM funds was not timely. However, the disbursements schedule specifics are not mentioned in the contract signed with FHI and the release of funds could take up to six weeks; • Documentation provided by GRIP to FHI 360 seems to have gone missing on a couple of occasions at the FHI 360 level. • The shift from AE to FHI 360 seems to have affected GRIP. GRIP used to receive financial TA more frequently and did not need to wait as long to receive the quarterly disbursements. • FHI conducts quarterly visits. GRIP staff goes to Pretoria often (once or twice a month) - though visits to Pretoria take GRIP field staff away from the projects in the field. The FHI 360 Program manager, the MER Officer and the Accountant Officer normally participate in these site visits. The benefit of such interaction is to talk out certain issues and receive fresh ideas. Capacity Building • When asked what Capacity Building means, GRIP staff indicated the following: (i) Building staff to reach the capacity to do the job; (ii) Empowering staff; Improving outputs to the survivors/delivery; Technical assistance/volunteer counselor become staff Sometimes, GRIP’s impression was too many trainings were offered (sometimes 2-3 days every month and all staff were required to participate) preventing staff from attending to their tasks and responsibilities. It would be better to have fewer trainings in the future (or shorter in duration, at least). Also, rather than generic trainings, it would more cost-effective to offer more specific training. FHI helped GRIP develop a CB plan (still in draft) and GRIP played an active role in designing it. Unfortunately, the organization is lagging behind with the implementation of the plan. FHI provided CB to GRIP in (i) Data Quality Assurance; (ii) Monitoring and evaluation; (iii) Organizational 142 South Africa Umbrella Grants Management (UGM) Performance Evaluation Governance (e.g., leadership, separation of governance and management functions, governance systems); (iv) Financial and Grants Management; (v) Resource Mobilization for Sustainability No CB was provided in terms of (i) Data Management, or (ii) Strategic Planning PEPFAR introduced funding of MARP activities. However, this was new to GRIP and they feel they did not receive adequate technical assistance to implement it (“it was no longer our program in a way”). The Most Common method for receiving CB was trainings for multiple partners - good for networking—but intense in terms of content—e.g., the SA NGOs in the room and newer staff invited to participate were exposed to difficult concepts in M&E. To empower staff, SPs were invited to send many employees but this results in many participants of different levels in the audience, not all of whom are able to benefit equally). To maximize benefits of participation, it would be ideal to tailor the level of the workshop content to participants in the future. The Most Effective method for receiving capacity building support was Customized, onsite training—(in the big group setting, the GRIP M&E officer did not understand much) External Technical Support GRIP also received TA from external consultants as part of the UGM partnership. This was related to the use and administration of the ETF (Education Training Fund). However, GRIP staff could not use it because the risk assessment by the three consultants was never delivered Service Delivery The UGM helped GRIP to meet or exceed program service delivery objectives. Examples: the set-up of care rooms, the number of people reached through the mobile clinics (250 instead of 25). They need to get better at providing HIV support groups for MARP. New services provided as a result of the UMG grant: MARP HIV support group (therefore, more population groups) – Asikuluma – Parental care; More districts; Quality of services partly improved (furniture and supplies definitely were an asset); Implementation was never interrupted MARP have been affected but not as highly and effectively as hoped (also due to their mobility) Defining and Measuring Success The third version of the M&E Plan is more in line with GRIP’s model. Instead of fitting into indicators provided by PEPFAR, GRIP came up with its own indicators. There is no specific indicator to measure capacity building/development within GRIP and no specific assistance provided to put them in place. 143 South Africa Umbrella Grants Management (UGM) Performance Evaluation SUSTAINABILITY GRIP has secured new funding as a result of UGM support. Messages on Social entrepreneurship sound challenging. GRIP says it is not a firm and does not have any product it is trying to “sell. “ GRIP worked on Sustainability plans but did not finalize them. The Board has responsibility for sustainability-related actions and it is for them to approve the plan (unfortunately, they have not read the plans yet). GRIP policies/activities are/were aligned with relevant national government plans and policies. GRIP shares part of the costs of the project funded by the UGM (20 % of cost for care rooms). In GRIP’s organization’s overall budget, UGM funds account for more than 70%. GRIP has established written guidelines or policies to guide operations in the following areas: • Financial Management (it needs to be updated) • HR Management • M&E • Strategic Plan Work Plan (current year) GRIP has written technical guidelines for programs, and/or training guidelines for the staff. These include Standards of understanding – training – key performance areas Innovation Best Practice: GRIP works with rape survivors and confidentiality. UGM Effectiveness Summary Index GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.2 2.25 2.8 2.4. 144 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: HOSPICE PALLIATIVE CARE ASSOCIATION Rationale for Conducting this Case Study HPCA worked under 2 different UGM mechanisms - Pact and AED/FHI360 - and under both grew in organizational size and capacity until it graduated. HPCA provides capacity building support for its members (SSP), and has mirrored much of the capacity building support received from its UGM partners in its training and mentoring model. This case study helps us learn how capacity building can effectively take root and cascade from UGM partner to sub partner to SSP. Quick Facts about HPCA • HPCA is the only association of member hospices in South Africa, with 184 hospice service delivery sites plus satellites as well as new affiliates in development phase. It mission is to build the capacity of members and partners to provide quality palliative care; it doesn’t directly implement patient care. All member hospices are funded by PEPFAR (and other donors) through HPCA. • HPCA defines itself as a capacity building and development organization. Its capacity building includes training and a large mentorship component delivered through 40 provincial palliative care development officers. Each member hospice is assessed according to accreditation standards developed by HPCA in partnership with the Council of Health Service Accreditation of South Africa (COHSASA) and a customized development plan is created. Sub-partner year of registration: 2004 UGM mechanism supporting the organization (+ year UGM support started): Registered under Pact first and then under AED; with Pact 2004 to 2007; with AED 2007 to 2009, signed an extension document in December 2009, graduated January 2010. Organization’ s Main programmatic area(s): Pediatric Care and Treatment, Care and Treatment, TB/HIV, OVC, Palliative Care (Basic Health Care) HBH, Palliative Care (TB/HIV) Capacity Building (Prevention, Treatment, Care and Support) Described above. Direct service delivery ((Prevention, Treatment, Care and Support)- Partners: member organizations, DOH, D of Corrections, DOE Target population: Primary: people in need of palliative care, both adults and children, and their family members in need of support Main geographical area(s) of intervention (districts and provinces): entire country Number of Staff: Had 50 staff members at end of 2009, now have 74. UGM Percentage contribution to the organization’s overall budget: PEPFAR funding comprises about 10% of overall budget. 145 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM Processes and Effects: HPCA Experience UGM Application Process and Pre-Award Assessment For HPCA, the application and funding process were rapid. Grants and Financial Management • Finance management from UGM partners deemed very helpful; stronger orientation program in first month to understand reporting and compliance would have been useful. Capacity Building • UGM partners provided capacity building for Finance, HR, M&E, annual reports; after graduating, HPCA missed it, had experienced it as a safety net. HPCA already had strong organization and development expertise, patient care expertise, advocacy skills; area least developed was M&E, also the UGM organizations had quite a strong focus on M&E. • Pact appeared to have more autonomy from Washington than AED, whose decisions had to be routed through Washington, sometimes causing delays. HPCA was not accustomed to time lags. At times the quality of responses was appreciated, especially with close out procedures since closeout with AED was bigger than with Pact; External Technical Support • HPCA did not receive external technical support from either of its UGM partners. Service Delivery • HPCA is not a direct service delivery organization, rather builds capacity of member hospice organizations, DOH, and other partners. Defining and Measuring Success • A feature in the new Hospice Data Management System allows every hospital to record any trainings it gives onsite. • HPCA has developed an M&E course for all field staff. • HPCA conducts an annual quality assessment evaluation using the African Palliative Care Association patient outcome scale. • HPCA uses a star rating system (Star 1 to Star 5) to score members based on standards leading to accreditation. HPCA developed standards with COHSASA as a milestone development process with 5 phases. Members achieving the 5 star rating move to the Maintain Accreditation Program (MAP). • HPCA is currently involved in the development of a pediatric outcome scale for Africa. 146 South Africa Umbrella Grants Management (UGM) Performance Evaluation Sustainability • PEPFAR funding helped HPCA to leverage other funding; “knowing we met USAID standards reassured other donors.” Anglo asked HPCA to manage funds for the health systems strengthening, play a UGM role with hospices for activities funded by Anglo. HPCA has provided capacity building support for proposal development, sponsored members to attend fundraising workshops, and developed sustainability guidelines that assess capacity not only for funding but also government, management, service delivery capacity. HPCA considers true sustainability to be the sustainability of quality palliative care, not necessarily the preservation of the organization. Its sustainability focus is on infusing palliative care into the main stream health sector. Innovation HPCA has restructured integrate its technical focus areas - palliative care, pediatrics, advocacy, education - and form provincial teams that align well with new NDOH PHC re-engineering strategies. HPCA conducts environmental scanning to assess capacity building opportunities that will help to strengthen palliative care at facility as well as community level. HPCA has developed a hospice data management system for M&E has presented it to USAID. It has been rolled out to 58 hospices across country and will roll out to remaining hospices this year. It used by hospice managers to manage patient care; when roll out is complete; hospices will be able to collect data for 90,000 patients per month. 147 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights KEY CHALLENGES • No longer having UGM partner as safety net for guidance when needed. STRENGTHS • Scale of capacity building for partners – in last 2 years HPCA has broadened its focus to include not only individual carers and member organizations, but also the health system within which organizations are located. • Growth of staff and low staff turnover; staff are passionate about what they do. • HPCA has use limited resources well, with staff mentoring and supporting hospices, and hospices cascading advocacy and training to other organizations. • HPCA is preparing itself to be accredited, has helped its members to become accredited, and now wants to develop a national standard for itself in conduction with COHSASA. • HPCA has a quality program across all departments, logging all incidents and striving for staff excellence. Bringing this good practice into training for member hospices. • Most programs are piloted and assessed before being rolled out. First involved in mentorship program in 1999 with seven member hospices, evaluated how effective mentorship was, now this is a standard practice. • Success of annual audits including USAID audit. AREA FOR IMPROVEMENT • Looking back, HPCA would have benefited from a forum where it could have learned more from the experience of other UGM partners and possibility from being paired with a treatment partner, from whom they could have learned more about how care and support complement treatment. LESSONS LEARNED AND PROMISING PRACTICES • Data management system monitors both facility based and home based care, aligns with DOH PHC e￾engineering strategy and some shifts in focus to home based services. • HPCA has a prisons project with Department of Correction Services, trains offenders to be care givers in prisons, now rolled out to seven prisons in hospice areas. • MOU with Department of Defense to train medical corps. • DOE is looking at HPCA’s project in Cape Town/Johannesburg that helps teachers to facilitate children’s bereavement, looking at how to roll this out in other schools. PARTNERSHIPS • Accreditation process partnership with COHSASA. • Palliative care trainings for DOH health workers. • Health system strengthening through health forums in districts to support shift in focus from NGO driven palliative care to palliative in all health facilities as well, and a more integrated national model. Documents reviewed: • HPCA Framework for Managing Sustainability 2011/12 • HPCA Good Practice Guide and Self- Assessment Tool for Managing Sustainability • Annual Report 2011 • Constitution • Most recent PEPFAR quarterly report • Monitoring Evaluation and Reporting Plan 2011-2012 148 South Africa Umbrella Grants Management (UGM) Performance Evaluation • http://www.hospicepalliativecaresa.co.za/ 149 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: HUMANA PEOPLE TO PEOPLE (HPP) Rationale for Conducting this Case Study HPP is seen as an interesting example of an international, non-US-based NGO which has received capacity building and technical support from a US-funded Umbrella Grant Mechanism, FHI360. HPP is very much community-oriented, and is known for its “Total Control of the Epidemic” or “TCE” approach, which it pioneered in Zimbabwe beginning in 2000. Also of interest was the transition of the project and partners from AED to FHI360 in mid-2011. The question was whether the transition was smooth for the UGM as well as the partners, and what if any effects or differences the partners observed. Quick Facts Sub-partner year of registration: 1995 as Section 21 Company 2003 as NPO UGM mechanism (+ year UGM support started): AED-FHI360 2007 (HPP reported it first received funds in 2005) Organization’ s main programmatic area(s): Direct Service Delivery – HIV Prevention (AB and Other Prevention), HCT, and Home-Based Care Target populations: PLWHA, families and individuals affected, infected, or at risk for HIV/AIDS Main geographical area(s) of intervention: Limpopo and Mpumalanga Number of Staff: 500 staff working on UGM funding 900 total staff working in SA UGM percentage contribution to the organization’s budget: 60% overall but 100% of UGM effort TCE organizes community members into quasi-military positions and structures (“troops” and “corps”) to canvas communities house by house and person by person. TCE provides HIV prevention messages to change behaviors, and has been replicated by HPP in many other countries. HPP has its origins in Denmark, and has been very active in HIV prevention in Africa for over a decade. UGM Processes and Effects: The HPP Experience: UGM Application Process and Pre-Award Assessment: HPP began with AED, which conducted a number of formal and comprehensive assessments of HPP covering all aspects of organizational development and technical components as well. HPP performed well on most components. Although it had its own systems, it did not have in place all of the detailed systems, policies, and procedures typically required for USAID funding. AED and HPP began to develop action plans to rectify any deficiencies and fill any gaps. Recall by HPP of the assessment and the many subsequent capacity building efforts by AED during our interview was not very clear or complete, however, compared to AED reports. 150 South Africa Umbrella Grants Management (UGM) Performance Evaluation Grant and Financial Management AED documents indicate a substantial planning effort and investment in provision of formal training and workshops to HPP and its staff. However, though the assessment process was participatory and consensual in nature, the adjustment to coping with the many requirements of USAID/PEPFAR as well as AED and FHI360 proved difficult for HPP. As a field-based organization, the bulk of whose agents come from the community, HPP was vocal about the difficulty of providing the level of detailed financial documentation required by the UGM, especially, original receipts. There is a strong impression of an organization that appreciates and values the USG resources it receives, but also longs for earlier days when life was simpler and accounting less complex. In addition, HPP expressed that it encountered ongoing difficulties with receiving timely disbursements from the UGM, which continued after the shift to FHI360. The UGM and HPP met with USAID to discuss the matter; FHI360 also documented financial management issues with HPP and even conducted a financial risk analysis. There appears to be room for improvement on both sides. The UGM could/should perhaps develop faster and more efficient mechanisms to review and approve financial reports and receipts – an online system using scanned receipts, for example, though this has been shown to be labor intensive initially for the partner, despite its long-term benefits for expense tracking. Also, while FHI360 has to scrutinize HPP financial reports and receipts, to avoid any unallowable costs, they should strive to balance this need with the need be rapid, responsive, and avoid overburdening HPP. The fact that FHI360 has to send financial reports to its HQ for review appears to add an unnecessary level of approval and delay. On the partner side, HPP will need to continue to develop its financial systems, polices, procedures and personnel in partnership with this and/or any future UGM mechanism. UGMs also have their own internal policies and procedures to ensure that they manage the risk associated with managing substantial grants to third party organizations, to avoid disallowable costs. Capacity Building AED and then FHI360 provided a substantial package of capacity building to HPP over the life of the project. HPP indicated general satisfaction with the capacity building and technical support, though they said that much of it was general in nature, and not tailored specifically to their needs. Also, HPP indicated that as a large international NGO, it already had its own systems in most areas – finance, HR, M&E, grants, etc. However, in most cases, to comply with both USAID/PEPFAR and the UGM, HPP had to adopt new methods or adapt its systems to be more compatible with those of USAID/AED. The amount of capacity building work in the beginning was described as overwhelming at times, to the extent that there was concern about interference with HPP’s field work. The rate of capacity building might need to be calibrated and better scheduled in the future so that it does not interfere with field work. HPP agreed they got a lot of support from FHI360 and enjoyed their relationship with the UGM, which they described as helpful and responsive. At the same time, they were honest about the fact that the UGM was demanding, and about the occasional frustration and headaches that they experienced first with AED and later with FHI 360. The greatest single frustration was the time required to approve their financial reports and disburse funds. 151 South Africa Umbrella Grants Management (UGM) Performance Evaluation External Technical Support HPP received external support to conduct a review of its salary structure and also to do RDQA. HPP received external support just last week. Service Delivery The situation of a large international NGO like Humana is different from that of a small local NGO. Humana felt that their HIV prevention technical approach, TCE, was well developed, and did not require much support from the UGM. On the other hand, HPP had less experience in other program areas such as counseling and testing and care and support. In these program areas, they felt that technical support from the UGM was more useful. Like many other large organizations, and some smaller ones, overall they felt it was mainly the amount of PEPFAR money rather than the assistance of the UGM that made the most difference in their ability to expand their services and reach new clients. Defining and Measuring Success (M&E) In this area, HPP had its own M&E systems prior to becoming a UGM partner. HPP received support from the UGM to effect changes that were required to record, aggregate, and report PEPFAR indicators in the required manner. The HPP program officer felt that they gained a lot by developing their M&E system and that they have better quality data now on which to assess the effectiveness of their programs, in terms of numbers of people accepting HCT or HBC. However, like other partners, they still cannot determine directly if they are successful in reducing the rate of HIV infection or not. This can only be determined periodically, via surveys such as the DHS. Sustainability HPP mentioned that UGM support and capacity building helped them develop stronger systems as an organization, which in turn facilitated their efforts to attract new donors such as the Global Fund. PEFPAR funds received through the UGM are only about 50% of their total funding as an organization in South Africa. HPP has its own private fundraising mechanisms as well. This diversity of funding, and the fact that HPP has developed “trademark” programs such as TCE, suggest that HPP has a significant degree of program sustainability. In addition to this, HPP draws all its prevention “troops” from the community, which may mean that there is a strong sense of local “ownership” for HPP programs than for some other NGO interventions. Innovation The HPP HIV prevention model for Total Control of the Epidemic (TCE) approach was established well over a decade ago. They have a new model called “HOPE” for HCT. 152 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights STRENGTHS • HPP is known for its TCE approach, which mobilizes large number of community members, and reaches large numbers of clients in the community. HPP establishes close, direct contact with the community. Social mobilization is a key strategy. HPP is highly visible in the community. With the addition of HCT and care and support services, HPP now has a better-rounded HIV/AIDS program. With the improvement of its internal systems and policies, HPP also is well-positioned now to pursue funding from other donors and the government. AREA OF IMPROVEMENT • HPP would like to see UGM financial disbursement systems become faster and more responsive in the future, though it is also appears that HPP needs to adapt and adjust more fully to the demands of large donors to attract and retain large-scale funding. HPP admitted that aspects of its M&E systems and financial management have improved, which allows them to be more professional across the board, and to utilize those capacities in their other programs, which in turn makes them more marketable. LESSONS LEARNED • There was a steep learning curve initially for HPP to become compliant with grant and financial management requirements of both USAID/PEPFAR and of AED-FHI360. It also required them to staff up for financial management and grant administration, both in terms of numbers and professional caliber of these staff. • There were and are ongoing delays in disbursements of funds to HPP, first from AED and now from FHI360, which continue long after HPP submits its financial reports. It is beyond the scope of the evaluation to determine conclusively whether the delays are due to errors or shortcomings on the part of HPP or the UGM, or both. However, it is clearly detrimental to the HPP or any other UGM partner program for the disbursement delays to continue. • Therefore, in the future, it would be in the interest of the UGM and partners to work diligently to identify and address causes of delay and dispute on both sides, in order to ensure that disbursements are timely and reliable. There is a risk of compounding amounts of financial risk being placed on UGM partners, in the sense that they must meet both USAID/PEPFAR requirements, as well as the requirements of UGMs, which seek to protect themselves against disallowable costs, as well as to prevent or detect any fraud as quickly and efficiently as they can. The need of UGMs to manage financial risks has to be balanced against the need to be as quick and efficient as possible in disbursing funds needed to keep partner programs moving. UGM Effectiveness Summary Index (1=below average; 2: average; 3: above average) Governance & Leadership Grant & Financial Management Service Delivery Capacity Sustainability 2.6 1.5* 2.2 2.6 Documents reviewed • 2010 Capacity Assessment • Technical Capacity Assessment (TCA) 2010 • TCE Financial Risk Assessment 2011 • Humana OVC TCA • Humana Pre-Award Assessment 2010 • COPs for 2007, 2008, 2009, and 2010: • Other assessments by AED and FHI360 153 South Africa Umbrella Grants Management (UGM) Performance Evaluation *Grant and financial management score is low due to HPP’s ongoing concerns about the review and approval of financial reports by the UGM, and disbursements delays 154 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: KHETH’IMPILO (“THINK LIFE”) Rationale for Conducting this Case Study This SP was previously part of ARK (Absolute Return for Kids), a UK based philanthropic organization. Pact actively assisted a seamless transition from ARK to Kheth’Impilo, established October 2009 with no interruption in patient services. Since inception, staff has grown by more than 100% to over 100 employees. This case study helps us learn how a UGMP can help a SP transition from depending on a foreign philanthropic organization to become a fully functioning indigenous organization. Quick Facts about Kheth’Impilo Sub-partner year of registration: October 2009 UGMP (+ year UGMP support started): Pact since 2005 (supported ARK and continued to support Kheth’Impilo) Organization’ s main program area(s): ART, PMTCT, HCT Partners: NDOH, PDOH, DDOH Target population(s): PLWHA, pregnant women, infants exposed to and living with HIV Main geographical area(s) of intervention: Operating in four districts of EC, KZN, WC and Mpumalanga and more than 90 sites Number of Staff: 1,028 UGMP % contribution to overall budget: 80% Kheth’Impilo is a South African NGO that supports the SAG HIV National Strategic plan through facilitating delivery of HIV and TB treatment, care and support services to adults and children through strategic partnerships with PDOH and government hospital and clinics and communities. UGM Processes and Effects: The Kheth’Impilo Experience UGM Application Process and Pre-Award Assessment The process started in 2005 with Pact, which was very helpful to Kheth’Impilo in managing the transition from Ark to SA NGO status in 2009. Grant Management • The disbursement of UGM funds has been timely, “last 18 months have been stellar”. • Pact visits at least quarterly, sometimes more often; there is frequent communication. 155 South Africa Umbrella Grants Management (UGM) Performance Evaluation Capacity Building • In early days, Pact was helpful in providing CB support for strengthening M&E systems. • Pact was not as helpful in Finance CB. Kheth’Impilo had developed an “environmentally friendly” electronic finance system. All supporting documents are scanned and reports can be generated quickly. Pact’s paper based finance system was not seen as helpful to them. • The UGMP templates for developing annual work plans and budgets have been useful for USAID and other proposal writing, budget development and project planning. • Over time, Pact has provided CB for data management, data quality assurance, M&E, organization governance, finance and grant management (focused on ensuring Kheth’Impilo complies with PEPFAR rules and regulations), fundraising • Pact was not as helpful with strengthening program service delivery; the USAID Activity Manager has worked on this aspect. • Kheth’Impilo is not aware of what Pact’s CB model is, but says that Pact invites organizations to workshops from time to time. • CB support has helped them meet USAID requirements for compliance and reporting. • CB should focus on organizational stability, growth, and future relevance to help them position themselves to access funds available within that environment. External Technical Support • Kheth’Impilo only received this from a consultant who led a fundraising workshop. Service Delivery • Kheth’Impilo conducted research with local universities to strengthen quality of services. Defining and Measuring Success • Kheth’Impilo assisted DOH to implement the ART MERL system, to ensure that all facilities collect high quality data. Sustainability • Kheth’Impilo is looking for private public partnerships that further socioeconomic development through Corporate Social Investment initiatives. Innovation In Human Resources, Kheth’Impilo; • Hired a skills development officer to do a skills audit for employees, identify skills and training gaps, and develop a database to track staff training. • Developed “learnerships” to build capacity of community workers as a possible career path. • Is gradually transitioning PEPFAR funded staff to government positions, as intended. Financial Systems: • Kheth’Impilo implemented systems to allow it to manage more money/people, and perceives that it now has capacity to function as a UGMP for smaller partners. 156 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights KEY CHALLENGES • There are differences between their electronic finance system and Pact’s paper based system STRENGTHS • (Kheth’Impilo indicated these may not be directly related to CB from Pact) • Electronic finance system • Alignment with government policies and priorities, • Accredited training programs • Linking eligible households with government grants • M&E, timely reporting, use of budget and work plan templates (linked with Pact) AREA FOR IMPROVEMENT • NA LESSONS LEARNED AND PROMISING PRACTICES • Staff development through skills audit, learnerships, promotions, and transition of NGO staff to government positions ENGAGEMENT WITH GOVERNMENT • Strong engagement with DOH at national, provincial and district levels, and with DHA and SASSA. • Aligned with National Strategic Plan for HIV/AIDS. • CEO and Clinical Support Cluster Manager sit on NDOH TWG group for PMTCT. • CEO worked closely with HIV Tech expert to the Deputy President, to edit the latest NSP. UGM Effectiveness Summary Index (1=below average; 2: average; 3: above average) GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.5 3 2 2.4 157 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: NATIONAL ASSOCIATION OF CHILD CARE WORKERS (NACCW) Rationale for Conducting this Case Study NACCW is a case study in sustainability for an entirely new national cadre of child and youth care workers. The Isibindi Project is a model of sustainable community based child care support because of its partnerships with provincial level government to fund the cadre of workers and community organizations to manage services; the model has already expanded to Zambia and is being considered as a possible model for Zimbabwe. Quick Facts about NACCW NACCW is a South Africa NPO that provides professional training to promote healthy child and youth development and improve standards of care and treatment for troubled children and youth at risk in family, community and residential group care settings. Operated as a social franchise, NACCW’s Isibindi model has been recognized by government as providing excellent cost-effective services to children living in HIV affected households. It focuses on deprived deep rural communities, primarily recruits and employees poor women to support vulnerable families. Project initially targeted child headed households, then expanded to granny headed households and vulnerable families where parents are sick and unable to care for children. The model is aligned to national policy and legislation, for both HIV and broader children’s rights and protection. 158 South Africa Umbrella Grants Management (UGM) Performance Evaluation Sub-partner year of registration: 1975 UGM mechanism supporting the organization (+ year UGM support started): With Pact since 2005. Organization’ s Main programmatic area(s): OVC Capacity Building (Prevention, Treatment, Care and Support) – Partnerships with member organizations and training of child and youth care workers. Direct service delivery ((Prevention, Treatment, Care and Support)- primary area: Orphans and Vulnerable Children OVC Through the Isibindi project, service delivery to affected children through community based care and support: support to household, provision of Safe Parks, support for children with disabilities, support for youth with substance abuse problems, empowerment workshops for young women and young men about their constitutional gender rights. Partners: Many grassroots CBOs, FBOs, NGOs working in rural communities affected by HIV. Target population Primary: children orphaned and made vulnerable by HIV (general population, youth, farmers, health professionals, PLWH, etc.,): households and communities affected by HIV Main geographical area(s) of intervention (districts and provinces): Number of Staff: NA Initially were serving 2,000 children a year, last year reported on 100,000 children, now looking at 6 times as many Isibindi sites. 32 districts and 51 municipalities and all 9 provinces. Currently 67 Isibindi projects. UGM Percentage contribution to the organization’s overall budget: 80% UGM Processes and Effects: NACCW Experience UGM Application Process and Pre-Award Assessment • Relationship with Pact started from a DSD recommendation that the Isibindi Project be considered for USAID funding. NACCW submitted a concept paper; Pact assisted its revision, and thereafter the Isibindi Project received funding approval. • Pre-award assessment focused on financial systems and adjustments that would need to be made, e.g. NACCW had to correct its payment of VAT, as a tax exempt NPO. Financial and Grant Management • Disbursements are very timely and there are frequent UGM partner visits, which always have an element of technical assistance, development and growth. • Without Pact’s direction, the Isibindi Project would not have adequate financial reporting systems to be able to work effectively in deep rural areas. They have used speed mail, and counter to counter post, a courier system. Capacity Building • Has been strong in all 5 Pact areas: Pact provided capacity building for development of annual plans and budgets; the rigor of this process helped NACCW, particularly for DSD 159 South Africa Umbrella Grants Management (UGM) Performance Evaluation funding, and provided an enhanced level of sophistication of thinking around budgets and implementation plans. • Pact developed a customized OD training for Isibindi’s SSP after visiting sites and meeting Isibindi network organizations with limited capacity. “Some partners say their capacity to raise funds has increased; we are transferring skills.” External Technical Support • Received from Pact when deciding whether to evaluate job descriptions to strengthen Board’s understanding about salaries. Pact helped NACCW set up contract with DeLoitte to rewrite JDs, evaluate and link them with salary remuneration packages to national standards, with appropriate percentile variances that come with NGOs. Believes that once we’ve done them we can share this learning with other organizations. Evaluating responsibility and complexity of jobs, • Pact put NACCW in touch with consultant who is very familiar with NGO governance to help them think through Section 21 aspirations. • Pact also provided a consultant to develop an Isibindi Safe Park site assessment check list to help standardize Safe Parks. Service Delivery • Rapid growth, organic evolution and expansion of services. Defining and Measuring Success Sustainability • The Isibindi Project is an outstanding example of sustainability in that NACCW is building a new skilled workforce, approved by national legislation, among unemployed community members. The DSD at provincial levels will commit to funding Isibindi project child and youth care workers; in this way a USAID funded project is gradually transitioning into a SA government funded project. First time this has happened at such scale in SA. Isibindi will oversee the roll out of 10,000 workers. Innovation • Several with the Isibindi Model: Safe Parks, disability program, empowerment program, substance abuse program, building skills of Isibindi Project mentors. 160 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights KEY CHALLENGES • Challenges attendant with working in poor rural areas, how to get receipts this level. STRENGTHS • Isibindi has evolved over 10 years, started as an idea, created a model, demonstrated it, profiled it, got government to see its value, integrated it into legislation with children’s act, government has decided it is sustainable and treasury will fund it. • NACCW’s purpose is to develop a work force for children in South Africa. It operates a large training program (14 core standards and specialized short courses), builds membership and leadership cadres, publishes information about child and youth care work to create and disseminate indigenous knowledge in our field, and develops advocacy models for children’s rights. • The project is building a skilled workforce among the unemployed, thereby responding to national poverty issues. • Linking with government, partners, communities, households, sharing the model and learning with other countries through USAID webinar series. AREA FOR IMPROVEMENT • MOUs with community level SSP to define relationships LESSONS LEARNED AND PROMISING PRACTICES • Safe Park model, which provides safe areas within communities where children can play and learn, has reduced gang violence in some localities as young men now have a place for constructive activities and opportunities to engage with male mentors. • Strong demand from communities for Isibindi Project model, has potential to become a stand-alone model that communities could develop through own initiative. • Children served through Isibindi are finishing high school; project helps them to access further education, through Safe Parks strengthening education components of children’s lives, not only playing but reading. • Disability program has exceeded expectations for recognition of inclusion; in some communities children who are physically disabled come to parks, have seen a transformation of way in which disability is seen. Other children helping them, what we hoped to achieve but has happened way beyond our expectations. UGM Effectiveness Summary Index (1=below average; 2: average; 3: above average) GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.67 3 3 2.4 Note: Pact UGM appears to have worked more closely with this sub partner across all 5 CB areas than any other Pact partner interviewed during evaluation. 161 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: NURTURING ORPHANS OF AIDS FOR HUMANITY (NOAH) Rationale for Conducting this Case Study Noah is a South African not-for-profit organization (NPO), conceptualized in 2000. Communities are the target of Noah’s interventions in that they are empowered through the establishment, or strengthening, of a network of care (or ark) to look after the orphans and vulnerable children (OVCs) within their community. The ultimate beneficiaries are the children. Noah currently has in total 100 Arks (also known as Child Care Forums) spread across the provinces of Kwa-Zulu Natal (67), Gauteng (28) and, most recently, North West (five). Arks are located in varying settings from rural through to urban areas. Forty (40) of these Arks are supported through PEPFAR funding (this will increase to 42 in COP11). At the core of every Ark are the committee and volunteers who ensure community support, manage activities and provide services to OVC. Quick Facts about Noah Sub-partner year of registration: UGM mechanism supporting Noah: Noah’s main programmatic area(s): Target population: Main geographical area(s) of intervention (districts and provinces): Number of Staff: UGM Percentage contribution to Noah’s overall budget: 2000 OVC Capacity Building (Prevention, Treatment, Care and Support) Human capacity development: NOAH trains volunteers, Ark staff and committee members' to identify OVC, conduct home visits, monitor OVC progress and link OVC to appropriate social services. Local organization capacity building. NOAH builds the capacity of community networks or Arks to support OVC affected by HIV and AIDS. Direct service delivery ((Prevention, Treatment, Care and Support) OVC care and support services: provided through resource centres /Arks and through home visits. Services provided include psychosocial support, basic health care, and HIV preventive care, legal and social assistance. Communities and OVC between ages 2 – 18 years. Total 4 650 targeted in COP11. Primary: Communities Arks are located in 13 districts in the provinces of Gauteng, North West and Kwa-Zulu Natal: Gauteng - City of Johannesburg Metro; City of Tshwane Metro; Ekurhuleni Metro; Wes Rand (Mogale City) North West, Moretele District KwaZulu Natal – Amajuba; Umkhanyakude; UThungulu; Umzinyathi; Umgungundlovu iLembe; Ethekwini Ugu Full-time staff: 286 (comprising 223 Ark staff and 63 admin staff 481 volunteers $3,9 million for 2009 - 2011 162 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM Processes and Effects: The Noah Experience UGM Application Process and Pre-Award Assessment • The organizational capacity assessment (OCA) was useful for Noah in identifying capacity gaps. Noah had two of these assessments since 2010 and has addressed most of the items. Karen doesn’t know what FHI does with the information. They don’t use it to help Noah develop a capacity building plan. Thinks that maybe FHI think that Noah don’t need the support because they are more developed that other sub-partners, and possibly other sub-partners have greater needs than Noah. • Would prefer it if FHI sat with Noah to work out what their unique capacity building needs are and then works out, with Noah, a plan on how to address these needs. Rather than assessing their capacity against a generic assessment tool. Grant Management • Noah received $3.9 million for 2009 - 2011 from USAID via FHI 360. • Disbursements from FHI 360 were not always timely. Very strict timelines have been challenges in that area, only challenge partners had said that turnaround time should be 5 business days, usually by end of month. Noah ends up funding out of their reserves and then waits to get reimbursed by FHI360. It is possible that FHI is doing reviews in batches because according to them Noah’s reporting is very good so the delay is not from their side. Capacity Building • Noah has participated in a number of external capacity building trainings. Some of it was useful, especially for the financial section, but it was too generic to be of any real value to the organization • Don’t have a clear understanding of the FHI 360 capacity building model • Twinning relationship with Heart Beat External Technical Support • External technical support was provided for an internal salary equity review. The effort was aimed at ensuring that Noah pays its staff members fairly according to market rates, and that compensation is equitable within the organization and maximizes staff retention. FHI 360 provided this technical assistance through the OM/HR consulting firm. The bench marking report included an implementation strategy so that Noah can affect the findings within their available resources. • Apart from the above external capacity FHI 360 has not responded to any requests for technical support for specific needs identified by Noah. • USAID Activity Manager was encouraged by another funder (MATT Foundation) to contribute to an OD exercise. Pact co-funded and contracted consultants for this exercise. : “FHI got a bit of a fright when they saw the advert in the paper – this was the first time they learned that USAID was involved in the OD exercise”. Noah had just assumed that they knew, Noah gives feedback to FHI 360 on this OD exercise at the quarterly progress meetings but she doesn’t send FHI any of the documentation – and FHI don’t ask for it. Intention was good. Communication was poor. Service Delivery • Noah has needed support in relation to programming but this has not been forthcoming from FHI 360. They have questions about how to best position themselves in relation to the Children’s Act – are they are a prevention and early intervention program, child protection, or both? What does this mean in terms of compliance with the Children’s Act? Should they register as a Child Protection Organization? • FHI 360 did an OVC Technical Capacity Assessment of Noah in June 2011. The tool is aimed at providing organizations with a set of criteria against which to assess their current technical capacity to implement quality HIV programs, to identify key areas that need to be strengthened, and to highlight strong project components that can serve as a model for other programs. For Noah however this was not a useful exercise because most of the activities in the plan of action that they had to develop as part of this assessment came straight from their strategic plan as they had already identified all the 163 South Africa Umbrella Grants Management (UGM) Performance Evaluation developmental areas before FHI 360 came to do the assessment. In addition, Noah never received any support to implement the recommendations after the assessment was done. Defining and Measuring Success • The Noah Model entails the development of Community Based Organizations (called Arks) to be able to function as independent and sustainable entities. Noah capacitates its Arks to move through a 4 stage development process towards independence. In terms of organizational development, the group of people who are interested in forming an ark are assisted in terms of forming a committee, and are subsequently trained on how to run effective meetings and keep minutes. From there, through a process of training and mentoring, the group eventually becomes an ‘ark’ via its progression along a predetermined growth path. There are currently four developmental stages in the Noah progression model. These are: 1. Establishment / Crawling; 2. Developing / Walking; 3. Maturing and Preparing for Partnership / Running; and 4. Independence / Winning. Independence means that the Ark will be able to adhere to sound administration policies and procedures, and have obtained a sustainable funding source. The South African Government (SAG) is seen as a very important role player in terms of funding the Arks once the independence stage has been achieved. Sustainability • Nothing that can be attributed to FHI 360. Noah has developed plans/strategies on their own or with support from other donors. In particular, they established the Noah Sustainability Trust. This is a for￾profit entity which creates endowment through investment activities. It is quite a process to set something like this up, Noah were lucky they got pro bono legal support. There are about 5 – 6 other NGOs who have set up similar Trusts e.g. Love Life. It is complicated to set these Trusts up, requires a business model approach to get it right. • Noah attended the FHI 360 sustainability workshops but never knew that they could have one to one sessions with Lisa (FHI sustainability consultant). They only found out this was possible when they had their quarterly meeting with the Activity Manager and FHI and USAID asked if Noah had met with Lisa to develop their sustainability plan. • Accessing pro-bono support for technical needs 164 South Africa Umbrella Grants Management (UGM) Performance Evaluation Highlights STRENGTHS • Support to financial section to comply with USAID requirements • Trainings for multiple partners – provided gateways for Noah to work with other partners • Noah described their relationship with FHI 360 as “very good…they are extremely nice people. They care about us. They see us as their ‘shining star”. AREA OF IMPROVEMENT • In-house or external technical expertise in OVC programming and the Children’s Sector • Tailor training to the needs of the organization. FHI don’t always follow-up on unique needs identified by Noah. They did this once (although it took a long time) – contracted an external consultant to conduct and employee salary benchmark exercise. • Don’t cluster developed sub-partners with emerging/developing sub-partners LESSONS LEARNED • Noah finds the site visits from FHI staff to look at and give input on systems more helpful that the generic training. UGM Effectiveness Summary Index (0=below average; 1: average; 2: above average) GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.8 3 1.4 2.6 Information sources Documents • OVC Technical Capacity Assessment Nurturing Orphans of AIDS for Humanity (Noah) 07 June 2011 • Noah financial risk assessment 2011 • Nurturing Orphans of Aids for Humanity (NOAH) Financial Risk Assessment Recommendations • AED-UGM 2010 Capacity Building Assessment Tool – Noah • Noah pre-award assessment report, January 2010 • AED UGM Annual Report October 1, 2007 – September 30, 2008 • AED UGM Annual Report October 1, 2008 –September 30, 2009 • AED UGM Annual Report October 1, 2009 –September 30, 2010 • FHI 360 UGM Annual Report, October 1, 2010 –September 30, 2011 • Noah Profile COP10 • Noah Annual Report. Draft Final Report – September 2011 Other • Interview with Noah team, 22 March 2012 • Interview with Noah Director, 30 March 2012 • Noah website: 165 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: SOUTH-TO-SOUTH PARTNERSHIP FOR COMPREHENSIVE FAMILY HIV CARE AND TREATMENT PROGRAM Rationale for Conducting this Case Study The South to South Partnership for Comprehensive Family HIV Care and Treatment Program (S2S) focuses on building pediatric care and treatment in South Africa. Before September 2010 South to South S2S had a sub-contract with ICAP/University of Columbia USA, also funded by USAID. When this contract ended S2S applied directly to USAID for funding but because the amount was less than $7 million per annum it was too small for USAID to manage and so they were put under Right to Care. (S2S annual budget is around $3.2). S2S is not a direct service provider but rather focuses on the development and implementation of District level quality improvement models for integrated Maternal (PMTCT), Child, & Psychosocial HIV & TB services. Technical assistance through training of heath care workers and partner trainers/mentors is provided using S2S developed tools and curricula. S2S is a USAID specialist partner in the response to specific clinical and health systems strengthening needs within South Africa. The focus and approach of the S2S program is very similar to that of FHI 360 Capable Partners which could also be considered a USAID specialist partner in the integration of nutrition and PMTCT but does not fall under an Umbrella Grants Mechanism (UGM). Quick Facts about South to South Sub-partner year of registration: S2S is a program of the Department of Pediatrics and Child Health, Faculty of Health Sciences, Stellenbosch University. The University of Stellenbosch is 100 years old. S2S was started in May 2006 as a partnership between the International Center for AIDS Care and Treatment Programs at Columbia University (ICAP) and Tygerberg Children Hospital at Stellenbosch University, with a focus on sub-Saharan Africa. The program was re￾launched launched in May 2008 to focus exclusively on the needs of South Africa. UGM mechanism supporting the South to South program: Right to Care started supporting the South to South program from September 2010. South to South program’s main programmatic area(s): S2S’s main programmatic area is Capacity Building (Prevention, Treatment, Care and Support) to promote the integration of a medical model of treatment with psychosocial support which has been identified as critical in providing on-going maternal care, increasing the uptake of children into pediatric care, strengthening adherence and promoting the health of the family. S2S aims to address existing health care worker knowledge, skills and system gaps by integrating and strengthening PMTCT, pediatric HIV and adherence and psychosocial (APS) programs in support of quality HIV care and treatment services. This is accomplished through: Onsite, district specific, technical and program capacity building activities which includes continuing and phased￾in skills building, knowledge transfer, supportive supervision, clinical mentoring and modelling to improve quality of care, strengthening clinical critical thinking/reasoning skills, supporting job-realignment and instituting a multidisciplinary approach to service provision. Off-site health care worker capacity building activities including the one-week paediatric HIV care & treatment training course based at the University of Stellenbosch Faculty of Health Sciences, Tygerberg Children’s Hospital. The aim of the program is to build paediatric HIV knowledge, training and mentorship capacity within USAID partner organisations and the DOH. The training philosophy is a mix of didactic learning with applied clinical learning at Tygerberg and Brooklyn Chest Hospitals. Technical assistance to the Department of Health and USAID implementing partners. S2S acts as the coordinator of the PEPFAR Partner PMTCT working group and hosts quarterly PMTCT partner workshops to discuss and evaluate various approaches to PMTCT program implementation. S2S also provides paediatric HIV technical assistance to the NDOH Maternal & Child Health and Integrated Management of Child Illness (IMCI) in revising training materials and toolkit development. Development and distribution of performance and training support tools and resources such as the S2S pediatric HIV Care and Treatment Toolkit for SA Health Care Workers, the PMTCT Toolkit; and Psychological and Adherence Counselling Support Training Toolkit & Facilitator Manual. Training materials are provided in lever arch files so they can easily be updated. 166 South Africa Umbrella Grants Management (UGM) Performance Evaluation Target population: Government and non-government clinical and management site staff. Main geographical area(s) of intervention (districts and provinces): Under COP 11 S2S supported 26 sites in 2 districts: Gauteng province - Tshwane district and North West province, Bojanala district. Off-site training programs are offered at Tygerberg Hospital, Cape Town in Western Cape province. Number of Staff: S2S employs 19 staff members, 10 of whom are involved in provision of services in the field. The program is divided into a Technical Unit and an Operations Unit. UGM percentage contribution to the South to South program’s overall budget: 100%. S2S received $3.5 in COP 2010 and $2.9 in COP 2012. UGM Processes and Effects: The (sub-partner’s name) Experience UGM Application Process And Pre-Award Assessment • S2S had a detailed pre-award assessment which included a visit from RTC staff for a question and answer session following which the S2S team completed detailed questionnaire. According to S2S no gaps or capacity building needs were identified, this is possibly because they were the ones who completed the questionnaire. From the perspective of S2S, they use the same financial and human resource management systems as the university which has a sound governance structure and in the 100 years of its existence it has never had a qualified audit. Grant Management • S2S have experienced on-going challenges with the way RTC requires them to report on expenditure including the reporting format, spreadsheets for financial reporting and supporting documentation. Most of these challenges seem to stem from the fact that S2S is not an NGO but are a program of Stellenbosch University which is regulated by the Higher Education Act not the Non-profit Organization Act. The sub-grant is signed between RTC and the University of Stellenbosch. It is a challenge for S2S to comply with the financial requirements of the University of Stellenbosch and RTC. As a result there have been a few instances of non-compliance. • A lot of time and energy has gone into reaching a common understanding between S2S and RTC on these issues. • RTC have under spent their budget in both COP 10 and COP 11 and this is due to staff resignations and long delays to fill key position thus having an impact on the travelling budget as well. Capacity Building • Overall, S2S benefitted most from RTC capacity building (mainly through off-site trainings) to enable them to comply with USAID financial and M&E reporting requirements. • S2S does not have a capacity building plan. They were not assisted to develop a capacity building plan by RTC nor have they developed their own capacity building plan. For S2S, capacity building is mostly about personal development through individual trainings. They have not given much thought to organizational capacity development. • S2S do not think that that RTC have the capacity to provide the type of technical assistance required by S2S. RTC are an implementing organization and S2S is a specialist technical service provider. Some of RTC staff attend the S2S training courses (so in effect S2S is building the technical capacity of the UGM staff). There has been some collaboration between S2S staff and RTC pediatricians but this can’t be said to have had any significant impact on S2S services. • For S2S the capacity building provided by RTC is limited to off-site trainings. They do not view the on-site visits or distance communication with RTC staff as a form of mentoring or coaching. S2S has a big mentoring component in their program e.g. nurse sits with another nurse and helps to fill in register or perform a procedure. For S2S mentoring is assisting a person to do something better than they did it before. When someone is mentored, they enter into an agreement with the person doing 167 South Africa Umbrella Grants Management (UGM) Performance Evaluation the mentoring. From the perspective of S2S, RTC does not do mentoring; their site visits are all about ensuring compliance and sorting out challenges with reporting, especially on the financial side. External Technical Support • The S2S budget/workplan makes provision for external consultants. RTC has not provided any technical assistance from external consultants over and above this and S2S were not aware that they could ask for additional external technical support over and above what they had budgeted for. Service Delivery • According to S2S, RTC did not assist the program in any way to improve the quality of their service delivery: “it was more a process where we were questioned about service delivery at the reporting stage (i.e. after the fact)”. • S2S did not receive any capacity building from RTC in clinical care/quality assessment. • Targets in some areas not achieved namely the participation of Health Care workers in the workshops (88%) and individual onsite mentoring follow up (42%). Reasons for these targets not being met was attributed to S2S staff shortages and being allocated fewer sites by the North-West province to implement their model. Defining and Measuring Success • S2S, as a capacity building program, do not have direct service delivery targets. S2S targets fall within the PEPFAR Health System Strengthening: Human Resources for Health category. - Program Area: PMTCT- Prevention and Care (Psychosocial and Adherence support) Pediatric - Treatment & Care (Psychosocial and Adherence support): Total number of capacity building events (training and mentoring) in the provision of family centers HIV services (Prevention of Mother to Child Transmission (PMTCT), Pediatric HIV Care & Treatment, Adherence and Psychosocial Support (APS), System Support); - Total number of HCWs trained and or mentored during events in the provision of family centers HIV services (Prevention of Mother to Child Transmission (PMTCT), Pediatric HIV Care & Treatment, Adherence and Psychosocial Support (APS), System Support) • Standards of Care Tools are used to assess sites at baseline and semi-annually thereafter to monitor progress made by S2S supported sites in the areas of 1) PMTCT at the antenatal and postnatal clinic; 2) PMTCT services provided at Labor and Delivery Wards; 3) Follow-up and Care of HIV Exposed Infants; 4) Care for HIV Infected Infants and Children; 5) ART Treatment Services for HIV Infected Infants and Children, and 5) TB/HIV Integration. • Outside of USAID indicator reporting requirements, S2S use DORA – a set of indicators developed by DOH for their Regional Training Centers and the DOH indicator dataset. This enables S2S to look at outcome indicators not only process indicators. STS also want to look at system indicators e.g. organizational culture of a system. Sustainability • S2S has only one source of funding: USAID via RTC. • The RTC pre-award assessment is very clear that S2S have no other sources of funding, and rely entirely on USAID for funding. However this was not identified as critical area for organizational capacity development. The only input from RTC in this regard was to invite S2S to attend the FHI 360 Sustainability Workshop in late 2011. • S2S were of the opinion that they developed a strategic plan to diversify their funding sources without any input from RTC: “RTC made us aware of the fact that we need to diversify our funding sources by arranging for us to attend a workshop. We were already aware of the fact that we have to diversify our funding sources a long time before the workshop and have been working on this without being prompted by RTC”. 168 South Africa Umbrella Grants Management (UGM) Performance Evaluation • S2S strategy for the next five years will be informed by the strategy of two grant applications currently in development. The strategy will continue to use technical training, mentoring, and DOH technical district support through systems strengthening using Quality Improvement methods. A new emphasis on operations research and innovation will ensure that best-practices are evidence-based and S2S programmers are formally evaluated. Innovation • No innovation in services or organizational development was attributed by S2S to RTC. • RightMax, which is an innovation from RTC in terms of financial management and reporting, has placed an even bigger burden on S2S: - “It was supposed to make submitting the supporting documentation easier but the S2S photocopier/printer does not have the correct software so they can’t upload the supporting documentation directly. They still have to scan and send to RTC who uploads and then sends back – still a lot of back and forth”. - “RightMax is very helpful for RTC – for S2S it is a lot of extra work”. The only advantage for S2S is that now they can prove that they did send supporting documents (in the past they would send supporting documents by mail and RTC would say they didn’t receive it). Highlights STRENGTHS • Timely disbursements of funds by RTC. AREA OF IMPROVEMENT • Would help if RTC could distinguish that S2S is not an NGO – it is a program of a 100 year old university which has never had a bad audit record and their financial management systems and human resource management systems are sound as well as their governance. • RTC reporting requirements should take into account the fact that the University of Stellenbosch has strict financial policies and rigorous annual audits: “the way reporting is currently done is placing a lot of extra work on the S2S financial staff because supporting documents (such as invoices) have to be submitted for each transaction on a monthly basis and then must also be explained to the RTC compliance officers one by one. For organizations like S2S with strict financial policies and rigorous annual audits in place this seems a bit over the top as it is taking up a lot of time”. LESSONS LEARNED • It was better when S2S was part of ICAP. The reporting was easier and they also had monthly calls on programmatic matters. Because they were part of a University, ICAP had the expertise to provide input to strengthen the S2S program – the whole set-up of S2S was done under the guidance of ICAP. RTC don’t have this kind of technical expertise, their staff members attend training programmers offered by S2S. • RTC are well-positioned to support other NGOs who are also implementing services, but do not have the technical expertise needed by S2S which is a specialist partner in the response to specific clinical and health systems strengthening needs within South Africa • “Overall, RTC are very nice people, they just micro-manage which makes it a challenge for S2S”. UGM Effectiveness Summary Index (0=below average; 1: average; 2: above average) GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 1.8 * 2 1.4* 2.2 *(this is low because S2S as a university program took issue with the UGM approach) 169 South Africa Umbrella Grants Management (UGM) Performance Evaluation Information sources Documents • S2S Pre-award Assessment Final 31 Mary 2010. • South to South Program for Comprehensive Family HIV Care and Treatment (S2S) COP11 – Annual Work Plan. • Right to Care NGO Quarterly Review Report Q3 COP 10 (April 11 – June 11): South to South. Prepared for USAID. • RTC NGO Sub-Recipient Quarterly Report: South to South, July – September 2011. Prepared for USAID. • South to South NGO Quarterly Review Report, Oct 11 – Dec 11. Prepared for Right to Care. Other • Interview with S2S Operations Director, Program Assistant and M&E Officer 28 March 2011 • South to South website: http://sun025.sun.ac.za/portal/page/portal/South_to_South/Index 170 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM EVALUATION CASE STUDY: WORLD VISION/SOUTH AFRICA (WV/SA) Rationale for Conducting this Case Study Supported by the UGM since 2007, World Vision/South Africa is a well-established organization that has been operating in several South African provinces for many years. Affiliated with the World Vision in the US, WV/SA was already benefiting from capacity building activities in several areas (e.g., service delivery, grants and financial management as well as organizational development) before entering the UGM program. Learning more about WV experience is likely to provide information on how to: (i) increase UGM added value when targeting a well-structured SP on the verge becoming a prime, and (ii) minimize risks of duplication of efforts as a way to enhance programmatic efficiency. Quick Facts about FPD UGM Processes and Effects: World Vision Experience: • UGM assigns SP financial and grants management responsibility and pushes all grantees to strive for both accountability and transparency in their day-to-day operations; • UGM is a good mechanism providing support and capacity building; • UGM fosters coordination between the donor, the UGM and the SP. In doing so, UGM strengthens systems; • UGM promote linkages with other partners (e.g. governmental agencies) Governance and Leadership • WV Board members meet more than once a year and their positions last for three years. Members can run for re-election for two more term. Board members who miss three consecutive meetings lose their status; • All Board members receive a board booklet handbook explaining decision-making processes and conflict mitigation procedures; Sub-partner year of registration 1975 UGM mechanism supporting the organization Pact since 2007 Organization’s Main programmatic area(s) We OVC (originally in 11 thematic areas – today in only 8) Capacity Building Relying on a well-developed grants and financial management training curriculum provided by WV HQ in the US Main geographical area(s) (districts and provinces) 19 programs in 6 provinces (5 districts). Number of Staff 314 Budget as a prime: $375,000 171 South Africa Umbrella Grants Management (UGM) Performance Evaluation • The UGM grant did not help WV-SA to become more rigorous – as attested by staff members - but rather to keep doing things right, in a rigorous and transparent manner; • Audits required for USAID grants were instrumental but are not an innovative activity. WV has its own internal audits and never hesitated to inform USAID of this; • The UGM enhanced WV discipline on financial matters. The WV procurement process was revamped and some serious reflection took place in the course of the site visits; • The WV job descriptions mention major responsibilities, knowledge, skills and abilities but they are not very specific on Key Performance Areas. Financial and Grants Management • UGM co-funds 6 programs in three provinces (five districts): Free State, Eastern Cape and Limpopo. Other funding is from WV of US, Canada, and UK and government of Malaysia. • Disbursements expected during the first quarter of the fiscal year (October-December) often get delayed due to time spent by WV and Pact on modifications to the contract; • Initially WV staff were to submit quarterly financial reports but then they started providing Pact with monthly reports. There was a discrepancy between the reporting schedule agreed with Pact and the parallel financial reporting set up for other donors. • It proved to be better to combine the quarterly and monthly reporting. The latter enhanced greater accuracy and better supervision of expenses among WV staff; • Expense and program review meetings take place twice a year. WV staff appreciate such meetings. They receive additional funding after showing progress. • The meetings provided a motivational boost: they put gentle pressure on sub-partners to deliver faster and better results. That extra effort sometimes makes the difference. • Also review meetings provide a venue for discussing areas of improvement. That explains why all the WV executives always attended. Capacity Building Model • WV already have a CB program. WV already provides its staff with grant development training and financial management training. It is more intensive and of longer duration than what Pact provided. WV staff write an entrance exam – then take a follow up test); • The Pact M&E training was simple and straightforward, it was a sort of refresher course for all those WV staff that had already taken the longer M&E course provided by WV HQ; • Only later, Pact started asking WV what capacity areas they would like to receive training in, concurrently to the finalization of the yearly operational plan; • WV staff have not yet received two courses: organizational governance and program design and implementation • Trainings for multiple partners have proved to enhance interaction and allow WV participants to share their experience. This was useful during the M&E training. The financial training also was effective, and was followed by personalized sessions; • Customized, onsite training was effective for more specific topics that required being better tailored to the context (e.g., the data base training) • Mentoring through either Site Visits, telephones and emails were very helpful as a follow-up to the training provided; 172 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Thanks to UGM support, WV now provides more psychological support to children. WV also refers children to social workers (DSD); • Pact staff showed versatility in providing support in all WV four areas of intervention: finance, M&E, programs and organizational development. Cost-sharing • The US grant accounts for 20% of the WV South Africa’s budget. Other resources were leveraged among private contributors; • Following the sustainability training, WV staff developed and submitted more funding proposals to different donors and governmental agencies. Highlights KEY CHALLENGES • Due to the disbursement delays, WV ends up receiving the money requested to cover activities for the whole year when there are only 8.5 months left. STRENGTHS • WV is well-known in communities where they implement • WV fosters community ownership; when they enter a community they stay there for at least 15 years. At the end of this period, they have capacitated a local organization • WV local staff always meet with the chiefs and community leaders AREA OF IMPROVEMENT • Modifications and obligations to contract for the next fiscal year should be regulated and settled earlier and on time LESSONS LEARNED AND PROMISING PRACTICES • See Strengths ENGAGEMENT WITH GOVERNMENT • Increasing HOW ALIGNED WITH GOVERNMENT PLANS AND PRIORITIES: • Consistent with its faith-based missions and values, WV supports all government activities espousing such principles as support of the most marginalized and integrity UGM Effectiveness Summary Index (0=below average; 1: average; 2: above average) GOVERNANCE & LEADERSHIP FINANCIAL & GRANT MANAGEMENT SERVICE DELIVERY SUSTAINABILITY 2.5 2.5 1.4 2.6 173 South Africa Umbrella Grants Management (UGM) Performance Evaluation UGM SUB-PARTNER CASE STUDY: WOZ’OBONA (“COME AND SEE”) Rationale for Conducting this Case Study From its beginnings in 1987, up to the present, Woz’obona has earned a solid reputation for excellence in early childhood development (ECD), one of the few organizations in South Africa to do so. However, prior to UGM support from FHI360, Woz’obona was still a small, fledgling service provider in the OVC field. This case study looks at how the FHI360/AED UGM capacity building helped to support the rapid growth and service expansion of a highly-specialized service provider focused in one specific geographical area. Quick Facts about Woz’obona Sub-partner year of registration: 1987 UGM mechanism (+ year UGM support started): FHI360-AED (2007) Organization’ s main programmatic area(s): -Direct service delivery (Prevention, Treatment, Care and Support) Direct Service Delivery – OVC care and support Target population: Pre-school to older children and their families or caregivers Main geographical area(s) of intervention: Limpopo for the OVC Safety Net Program Number of Staff: Staff in Limpopo: 7 Staff in Pretoria: 1 UGM percentage contribution to the organization’s budget: 100% of OVC program Woz’obona is an interesting case study, with a dynamic Director, who is clearly passionate about the mission and work of the organization, and staff who seem to share the Director’s dedication and passion for the children they serve, their families, and their pre-school educators drawn from the local community. Their basic philosophy, “follow the child,” leads them to believe that they are effective in discovering the true needs of the child and family or caregivers, rather than simply implementing a pre-determined package of interventions prescribed from outside. UGM Processes and Effects UGM Application Process and Pre-Award Assessment AED conducted the initial UGM pre-award assessment in 2007. At that time, Woz’obona was a small, specialized NGO, focused on early childhood development (ECD), localized in one site in Limpopo province (Jane Furse). Woz’obona reports that it was previously affiliated with the Nelson Mandela Children’s Foundation, and did not apply to USAID through the APS process. They indicate USAID selected and assigned to AED as their UGM, presumably to replace NMCF funding. 174 South Africa Umbrella Grants Management (UGM) Performance Evaluation Their initial funding was for OVC work, and was approximately four times as much as their budget at that point. It was seen as a windfall. According to the Director, the AED pre-award assessment was thorough in terms of grant/financial management, HR, and M&E, governance, and other organizational areas. Findings by AED indicated that Woz’obona had a number of strengths but also had deficiencies as compared to USAID/PEPFAR reporting requirements, rules and regulations, and standards for financial management, HR, and governance. Woz’obona acknowledged the concerns and developed a plan to begin rectifying deficiencies and filling organizational gaps. Grant and Financial Management The UGM, first AED and then FHI360, worked extensively with Woz’obona on improving its financial management. Grant management for sub-awards was not required as Woz’obona was not making sub￾grants to other organizations. Woz’obona also worked on improving management of its own award. Prior to the award, Woz’obona recalls that it had a simple, cash-based accounting system, which it had to convert to a USG equivalent financial management system. This involved significant work initially, but Woz’obona now says it makes them stronger too, and also positions them to pursue funding from other donors, confident that they can meet any donor reporting and financial management requirements. The improvements made in many areas significantly reduced financial risks, through institution of safeguards in terms of accounting, cash management, asset management, HR contracts and payroll procedures. In general, Woz’obona described a good relationship with the UGM, first AED, then with FHI360. Communication and collaboration were described as positive and helpful. However, as with other partners, Woz’obona described the initial period as challenging, commented on the steep learning curve, and on the demands of compliance with USG and UGM systems. Also, as with other partners, Woz’obona observed that UGM demands at times came on short notice, and time allowed for responses was perceived as inadequate. Organizational Capacity Building AED and FHI360 documents describe their initial and ongoing work with Woz’obona and the resulting improvements in many areas, such as financial management, project planning, HR, and M&E. On the other hand, over time, there remained some concerns in areas such as sustainability, mentoring/coaching, and governance. Woz’obona continues to define capacity building as being primarily about people, rather than about the organization. The UGMP CB model was not clear to them. Of interest were reciprocal assessments of capacity building (CB), for example in 2009, in which the UGM allowed the partner to rate the capacity building it received. Overall, Woz’obona rated all segments of the CB highly, but in specific areas, such as governance, there were indications that the CB did not respond to or meet the partner’s expectations. The Governance area posed challenges initially, in terms of needing to minute meetings, and to hold the meetings regularly. However, these concerns were addressed promptly. External Technical Support The main examples of external support cited included development of the HR manual, the implementation of audits, and strategic planning. Service Delivery Support 175 South Africa Umbrella Grants Management (UGM) Performance Evaluation Woz’obona first indicated that the UGM does not assist with service delivery, i.e., ECD, which is seen as being outside the UGM technical area of expertise. This appeared to be a narrow definition of service delivery and the ways in which the UGM assisted Woz’obona with OVC service delivery. Discussion resulted in acknowledgement that UGM support was useful in a number of ways: funding, mobility, improved organizational systems, policies, and procedures, M&E, and donor reporting. The massive infusion of USAID funding allowed Woz’obona to expand its geographic coverage, its client population, the number and variety of OVC services it offers, and also to improve quality, at least in the sense of improved supervision and training of community care volunteers. Still, it seems fair to say that Woz’obona remains the driving force in terms of the ECD vision, program design, and underlying philosophy: Follow the child. Defining And Measuring Success Woz’obona received significant UGM assistance to improve its M&E capability in order to track and report numbers of OVC clients, first to AED and then FHI and ultimately to USAID. However, discussions indicate that Woz’obona already had a strong M&E culture when it became a UGM partner. Statements suggested that Woz’obona takes data collection and analysis very seriously, a good indication for a small NGO. According to Woz’obona, they had initial concerns about the PEPFAR OVC indicators, which did not capture many key aspects of Woz’obona’s work with children. This required a shift in focus onto numbers of children reached and caregivers trained. Woz’obona acknowledged that the UGM provided CB in specific aspects of M&E, such as Data Quality Assurance, data management, and M&E. However, in the remote rural town where Woz’obona works, it may be difficult to attract and retain highly-skilled M&E staff, which are in demand everywhere. Identifying pre-qualified staff locally appears to be a challenge. Woz’obona arranged to train existing staff in M&E. It is too soon to say how successful this approach will be. The UGM may be able to help build M&E capacity. Sustainability This remains an area of concern for Woz’obona, whose UGM funding is 100% of the cost of the OVC program per se. Woz’obona appears focused primarily on its core mission, ECD, and according to CB assessment documents, may not be able to allocate adequate time and energy to fund-raising to support the OVC program in Jane Furse after the UGM ends, unless additional OVC funding is forthcoming under a future mechanism. On the other hand, Woz’obona has had some fund-raising successes in other areas of operation. They shared news of a large primary education program recently funded by the Department of Education in Gauteng. However, the two programs are not related; they are distinct geographically and programmatically. It is possible that UGM support contributed to their ability as an organization to develop this proposal and to win the award. Other sustainability concerns include: succession planning in the event the director should wish to retire eventually, after many years of service to Woz’obona; and decisions about whether to bid (or not) on other major grants from government or donors. Woz’obona expressed that, like many other small organizations, it often needs help to understand and assess the risks and benefits in pursuing various strategic courses of action. The UGM was seen as better at providing standardized packages of CB to NGOs, in operational areas such as grant and financial management, HR, and M&E, and less able to assist partners in these kinds of strategic, sensitive, and highly individual organizational concerns. 176 South Africa Umbrella Grants Management (UGM) Performance Evaluation Discussion led to identification of one option, namely, to arrange for retired executives with relevant experience to mentor and coach NGO Directors or executives, and their boards, to work through key decisions and strategic issues. UGMP may not be staffed with adequate executive talent to coach NGO leaders and boards through these kinds of issues. Innovation Woz’obona is recognized as an innovator and technical leader in the field of ECD, although this is just one of the primary foci of OVC work. Highlights STRENGTHS Woz’obona is recognized as an outstanding provider of ECD/OVC services. It is also a very good example of a small and localized but very skilled technical service provider which benefitted significantly from the strong support of its UGM partner, AED and then FHI 360. Woz’obona learned how to manage USG/PEPFAR funds, how to handle PEPFAR reporting, and how to track and report PEPFAR indicators. Disbursements from AED and then FHI360 were described as very timely upon receipt of monthly financial reports. UGM progress reports and assessments indicated that Woz’obona has improved significantly in many areas of capacity building, suggesting two things: 1) the UGMP has been effective in providing quality capacity building to the partner; and 2) the partner has been receptive and committed to improving. AREA OF IMPROVEMENT • Woz’obona commented that it has professionalized many areas of its operations thanks to PEPFAR investments and FHI360 capacity building, such as financial management, M&E, HR, and grant management. However, it recognizes the need to improve in other areas, such as governance, strategic planning, and sustainability. The improvements resulting from the UGM support have helped to empower Woz’obona to pursue funding successfully from other donors, such as the Department of Education. Woz’obona has a strong mission and commitment to ECD, but would benefit from continued organizational capacity building to raise funds, make strategic plans and decisions, and improve its sustainability. • The areas for improvement applicable to the UGM include: providing better and more complete information to the partners about the UGM capacity building (CB) approach and plans; scheduling CB activities with partners well in advance, and taking into consideration the partner’s own activities and work plan, so that CB does not interfere with the core mission; giving more advance notice when making requests of partners; as well as allowing, whenever possible, more time for partners to respond to requests for information and action. LESSONS LEARNED 177 South Africa Umbrella Grants Management (UGM) Performance Evaluation • Woz’obona also benefitted from strategic planning exercises. As with other UGM partners, there was a steep learning curve initially, regarding USG and PEPFAR reporting and other requirements. Once the partner had begun to master these skills with help from the UGM, they found that the funding and capacity building was a significant net gain for their organization. • Over time, the partner’s need for capacity building support from FHI360 has decreased, which might mean that the UGMP could re-allocate some of its funding and staff to other, newer partners who need more help, or adjust the CB package that it now offers to Woz’obona. • Ultimately, if they wish to continue receiving USG funding, NGOs like Woz’obona may have to adjust to the kinds of high pressure, frequent demands, and short time frames that come with large scale USG funding, in particular, from PEPFAR. To do that, it might be advisable for Woz’obona to view this situation as the “new normal,” expect it, and plan for it. This kind of mind-shift has implications for many aspects of their work: their internal planning processes, HR issues like staffing and job descriptions and team-building, and on a higher level, for leadership, strategic planning, and M&E. It may even reach up to board development level. • A further observation is that the UGMP in the future may continue to struggle to comply with these partner requests, to the extent that PEPFAR and USAID make requests to them, with short fuses, and they have no alternative but to give the partner even shorter due dates. However, where possible, the UGMP should scrupulously plan their schedules with the partners well in advance, and make clear when short turnaround requests are inevitable. • An observation for partners of the UGMP in the future is that they should be well-briefed at the outset about the demanding nature of compliance with USG rules and regulations, and the high pressure and quick turnaround times involved when one accepts large scale USG funding. • SP would be well-advised to plan to staff up for PEPFAR, and perhaps to build adequate staff structures in when they respond to a call for bids issued by the USG, rather than try to take an overly lean staffing approach as a means to increase their chances of winning USG funding. • Ultimately, if USAID, the UGM(s) and the partners are all on the same page in the future about what is expected and required, the collaboration can be expected to improve, as will the levels of performance and results. Conversely, to the extent that partners continue to be somewhat in the dark or surprised about expectations, time frames, pressure, and so on, there will be a loss of opportunities, and possibly of trust and good will at various times and at various levels. UGM Effectiveness Summary Index (0=below average; 1: average; 2: above average) Governance & Leadership Grant & Financial Management Service Delivery Capacity Sustainability 2.6 2.5 2.4 2.3 Documents reviewed: • Interviews with Woz’obona in Pretoria and Jane Furse • COPS for 2007-2010 • UGM assessment documents from pre-award to present • Financial risk assessment, 2011 • Annual reports 2007-2011 • Work plans, 2007-2011 178 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX G. FHI 360 CAPABLE PARTNERS (CAP) ANALYSIS OVERVIEW OF CAP CAP is a Leader with Associate (LWA) Award. As the evaluation focuses heavily on the three UGMs, the SI Team elected to develop a more in-depth case study on CAP in order to ensure that it received adequate attention in this evaluation report. This section provides a detailed analysis of the CAP program. Other findings for the CAP program are limited to a few paragraphs in the External TA section. Since 2004, USAID/SA has provided TA to the SAG for improving infant and young child feeding, with a special focus on optimal nutrition in the context of HIV. Initially this support was provided through the LINKAGES project. Following the close of the LINKAGES Project in June of 2006, USAID/SA executed an FHI360 CAP Associate Award in October 2006 to ensure the continuation of this technical support through the Leader with Associates Cooperative Agreement between USAID and FHI 360. Since 2006, CAP has provided TA for priority district sites in all nine provinces at the request of provincial DOH Directorates of HIV, MCWH and Nutrition. TA is provided through CAP’s integrated model of capacity strengthening, which adheres to the WHO building blocks for health systems strengthening and fully integrates PMTCT, maternal health and IYCF of HIV service delivery at health care facilities and community services. In 2010, during the joint PEPFAR-SAG national and provincial realignment meetings for PEPFAR partners, the role of CAP (then AED) was reinforced by naming CAP a specialized Provincial PEPFAR partner with the ongoing role of providing TA for training and development of guidelines and tools related to the integrated PMTCT and nutrition model. At annual strategic planning meetings, held by CAP in conjunction with USAID, NDOH and PDOH representatives, requests for TA from CAP are presented by Provincial DOH leaders. Following the planning meeting, CAP designs its annual work plan, which USAID approves. Features of CAP’s integrated TA model include the following: 1. National, provincial, district, sub-district, site, and community advocacy for integrated PMTCT programming; 2. TA to the DOH for policy, guidelines and national training curriculum; 3. Training facility and community health providers in PMTCT technical state-of- the-art, counseling and negotiation skills, and social and behavior change communication (SBCC); 4. Introducing an eight-step PMTCT program on-site implementation process using site data, national policies, best practices and site-specific challenges to design an integrated program in conjunction with health facility and community providers; and 5. Providing supportive follow-up with monitoring and evaluation for sustainability and using results and lessons learned for program improvements. The model also uses the Baby-Friendly Hospital Initiative (BFHI) as a strategy to ensure the quality of MCWH services at facility level (e.g., PMTCT Mother Friendly care and Better Birth Initiative) and mobilizes NGOs and communities to influence social norms and adoption of optimal reproductive health, PMTCT and IYCF behaviors in the community. 179 South Africa Umbrella Grants Management (UGM) Performance Evaluation CAP currently employs nine staff members, including six technical staff (one of whom focuses on M&E), two administrative staff, and one financial person, representing moderate growth from six staff members in 2006. CAP’s budget allocation from USAID for 2006 – 2012 was $8,742,249. Extent of CAP progress towards the achievement of its CB goals and objectives The CAP TA program to DOH has nine objectives, all of which relate to the delivery of integrated PMTCT and nutrition services. This section examines the extent to which progress has been made in achieving these objectives. (1) Assist NGO and government health-care and community facilities in selected sites to design, effectively implement, monitor and evaluate integrated PMTCT program in order to sustain implementation of integrated programming interventions and reporting of results. In 2006, CAP was given a target of 250 sites to support. The organization tailored their CB model (described above) to enable them to reach the target with limited budget and staff. The approach under this objective is to offer on-site training to health workers, managers and community caregivers from one sub-district. The sub-district will have been identified by the provincial DOH as a high priority for various reasons, e.g. under-performance. During the training, participants develop plans for implementing actions related to the integration of PMTCT and nutrition in their facilities. Following this training, one health facility in the sub-district is identified as a central hub/demonstration site. CAP staff visit the central hub/demonstration site over a period of months to demonstrate what they mean by “integration”. During the visits they are available for on￾site mentoring and coaching of health workers and community caregivers in the implementation of their plan. This on-site mentoring and coaching builds and supports health workers’ capacity to plan, implement, monitor and evaluate their programs and activities. It encourages their adoption of key service delivery reforms that will improve maternal health and HIV free child survival. In addition, their own PMTCT data is used by CAP during the on-site strengthening process to help them better understand the importance of data in improving service delivery. Health care workers from the other facilities in the sub-district are invited to learn from the demonstration site and replicate good practices in their own settings. Through the implementation of this model, CAP has successfully met its target of 250 sites over the six year period. This is, however, a cumulative number and not all sites are currently active. There are currently seven active hub sites and 110 facilities in the sub-district, also called feeder clinics. (2) Build capacity of health care workers at facility and community levels in the integrated program. In addition to the on-site/facility coaching and mentoring support described above, CAP also provides off-site training to health care workers and community caregivers from the same district/sub-district. 180 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 1: Analysis of Training and Participant Numbers 2006 - 2011 PROVINCES – NUMBERS OF PARTICIPANTS NO. AND % OF PARTICIPANTS TRAINED BY COURSE EC FS GA KZN LIM MP NW NC WC 1-day Decision Makers Course 0 0 0 0 29 98 0 106 0 233 (5%) 3-day Abridged Integrated PMTCT Training for Doctors and Senior Managers 14 32 42 60 7 3 4 9 30 201 (4.3%) 3-day Abridged Integrated PMTCT Training for Implementers 0 0 0 0 0 28 0 13 369 410 (8.8%) 5-day Abridged Integrated PMTCT Training 318 8 154 447 127 136 10 368 723 2291 (48.9%) 10-day Integrated PMTCT Training (Core Course) 21 49 51 97 58 30 49 23 101 479 (10.2%) 5-day BFHI Assessors Training 27 24 65 53 47 66 62 47 82 473 (10.1%) 4-day SBCC Course 0 0 0 28 0 65 15 49 31 188 (4%) On-site Mentoring and Coaching 0 0 0 47 0 0 0 30 17 94 (2%) 1-day Workshop for Strengthening Clinic Committees 0 0 0 0 0 0 0 0 0 0 Advocacy skills development workshop 0 0 0 0 0 0 0 0 59 59 (1.3%) ENA 20 0 0 0 0 0 0 0 0 20 (0.4%) RAP Assessment 0 0 28 94 0 0 0 58 30 210 (4.5%) Code Workshop 0 0 0 25 0 0 0 0 0 25 (0.5%) Total 400 (8.5%) 113 (2.4%) 340 (7.3%) 851 (18.2%) 268 (5.7%) 426 (9.1%) 140 (3%) 703 (15%) 1442 (30.8) 4683 (100%) (3) Strengthen facility-community referrals and linkages and use state-of-the-art social and behavior change communication (SBCC) strategies and approaches to influence social norms and adaptation of optimal reproductive health, PMTCT and IVCF behaviors in the community. Between 2006 and 2011, CAP trained 4,683 health care workers and community caregivers. The majority of trainings (72.2%) focused on the integrated program and included either a 3 day abridged course, a 5 day abridged course, or a 10 day core course (see Table 1 above). One of the main criteria for the selection of provinces and districts/sub-districts for TA should be alignment with the 18 priority health districts; ideally CAP TA should be concentrated in these 181 South Africa Umbrella Grants Management (UGM) Performance Evaluation priority (worst-performing) districts. However, the provincial spread of trainings does not match the provincial spread of priority districts. Eastern Cape has the highest number of priority districts (six) but only received 8.5% of the training; while the Western Cape has one priority district, but received 30.8% of trainings. KwaZulu Natal has four priority districts and received 18% of training. A key deciding factor for training appears to be the ability of the province to cover the travel and accommodation costs of participants as well as contribute to cost-sharing. WC was seemingly the only province that was consistently able to meet their financial obligations. In other provinces, DOH funding challenges resulted in cancellation of trainings at short notice. The ability of most provincial DOH to contribute to CAP training costs is limited. Provinces have limited training budgets, if any, so are unable to pay for an external training provider. Some provinces struggle to assist with participant transport and accommodation costs. Only one province, Western Cape, covers the costs of the training venue and catering. For most provinces, the training offered by CAP is a valued addition to their existing training program. Provinces would not be able to reach as many health workers if they did not have the external (and free) training support provided by CAP at facility level, CAP has also focused on strengthening of community clinics and involving community caregivers in behavior change and communication training. An example of the successful involvement of community members is provided in the box below: In KZN the contribution of CAP to CB of community caregivers was viewed by the province as an unexpected yet very welcome addition to the scope of CAP’s work. In KZN there are 10,000 community health workers who need to be trained so they can participate effectively in the multi￾disciplinary team. KZN saw a clear role for CAP to focus on the training/up-skilling and supervision of this cadre of workers. One of the challenges CAP experiences in training community caregivers is that they “don’t move as fast in their capacity evolution as health workers” (CAP respondent). This means that they require more intensive and longer-term CB support. However, the investment appears to be worthwhile, as each caregiver is responsible for tracking mothers in least 20 households and ensuring that these mothers and their babies return to the clinics for follow-up visits and on-going treatment/support. (4) Promote the revised Baby Friendly Hospital Initiative (BFHI) as a guiding and quality improvement strategy for optimal ICYF in the context of HIV, Mother Friendly Care and PMTCT in the integrated program. INVOLVING CLINIC COMMITTEES IN SERVICE DELIVERY: CAP SUCCESS STORY The District is in a deep rural area and transport options are limited. The clinic opened at 8 am and closed at 430 pm, and community members were complaining about these hours. Community members working with the nurses through the clinic committee came to realize that this was not just because of the attitude of nurses, it was government policy. The community and clinic developed a monitoring tool to monitor attendance of pregnant teenagers and found that their attendance was very low. They had to be in school so couldn’t come to the clinic at 8 am. Working clients also couldn’t get to the clinic in these hours. As a result, pregnant women were coming in for a first ANC visit at 38 weeks; for PMTCT to be successful women need to come in by 14 weeks so they can be initiated on HAART if eligible. The clinic committee asked themselves how they could go beyond the government policy so that the clinic could be flexible in responding to client needs. They negotiated with the clinic sister to extend clinic hours. This saw the introduction of early hours and flexi-time for both school children, and working clients who knock off at 4pm – 5pm. Attendance has subsequently gone from below 30% to 75%. People understand that departmental policies are there, but need to be adaptable to the local conditions. 182 South Africa Umbrella Grants Management (UGM) Performance Evaluation CAP participates in a number of national and provincial BFHI forums and provides TA to health facilities on the application of BFHI tools, including the BFHI self-appraisal using the standardized (WHO) self-assessment tool. (5) Train BFHI assessors to use hospital self-appraisal tools and global criteria to ensure quality of MCWH services at facility level. CAP is the only organization training Baby-Friendly Hospital Initiative (BFHI) assessors both at national and provincial levels. BFHI assessor training has been undertaken in all provinces. In Western Cape, the BFHI assessor training was seen to have contributed to the significant increase in BFHI accredited hospitals (from 2 hospitals to approximately 19). (6) Assess the quality of services and care provided through monitoring and evaluation activities (evidence research). The on-site coaching and mentoring model has been identified by DOH as a positive practice with visible results, particularly at the central hubs. However, national and provincial DOHs identified a need for more research-based evidence on how, and to what extent, the CAP CB model is contributing to positive outcomes that have been observed in most of their sites. Baseline studies have been undertaken in most of the sites and follow-up impact evaluations are scheduled for 2012 which may provide DOH with the information required. (7) Expand the scale of activities within targeted districts and sub-districts. CAP- assisted sites grew from 83 sites in six sub-districts in four provinces in 2007, to 243 sites in eight sub-districts in six provinces in 2009. In 2011, CAP was still working in eight sub-districts in six provinces, so the number of sites has not grown since 2009. CAP funding and capacity constraints are contributing factors to this lack of further expansion, A frequent request from DOH to the ET was for CAP to extend this on-site support to other non￾performing sites. Some DOH officials reported a “ripple effect” of improvement to other sites that learned from the improved practices of the CAP site. However, this was not the case in all sub￾districts. It was also observed that in some cases, an unintended consequence of strengthening a sub￾district facility was that it was then over-burdened with other new projects/initiatives, which ultimately weakened the facility and undermined the improvements brought about by CAP. (8) Work more concertedly to train and build capacity of Regional Training Centers based on the new DHO directive – Training Trainers. CAP has had limited success in strengthening the Regional Training Centers through a train-the￾trainer program, as RTC are currently not sufficiently resourced to take on this role. Provincial health workers have been trained in the 10-day integrated PMTCT training (core course) which should capacitate them to train other health workers in the 5-day integrated PMTCT training. However, DOH reported that in most instances, these workers don’t have the time to train others and in many cases don’t have the necessary facilitation skills (this is an area not adequately covered in the 10-day training). Even if they were able to do the training, they would not have the time to do the on-site mentoring, which is such an important feature of this model. There is high staff turnover within Districts/sub-districts and facilities. This means that training in the integrated PMTCT course needs to be ongoing. Until DOH strengthens the Regional Training 183 South Africa Umbrella Grants Management (UGM) Performance Evaluation Centers they are likely to rely on external specialist service providers, such as CAP, to provide this type of training, and if the training is offered at minimal cost, all the better. There are many competing demands for health budgets, and capacity building/training tends not to receive the priority it requires. Provincial DOHs are allocated budgets for operations and this includes training, therefore provinces are responsible for organizing and funding training. There is however an imbalance between the provinces in terms of their planning of activities and management of budgets. Provincial DOHs could potentially take over the off-site training if/when Regional Training Centers have been sufficiently developed, but they would still require on-going external TA for the on-site coaching and mentoring. (9) Provide guidance and mentorship to treatment, care and support implementing partners (IP) to integrate food and nutrition services into their existing programs using the USA IS/PEPFAR Nutritional Assessment and Counselling Support (NACS) approach This is a new CAP objective. No activities relating to this objective were specifically identified by DOH. Analyze key activities, recommend those that need to be modified or enhanced Annual CAP Strategic Planning Each year, CAP hosts a national strategic planning workshop in September for national and provincial DOH officials in the MCWH, HIV and AIDS, and Nutrition Directorates. USAID representatives also attend. At these annual strategic planning meetings requests are made by provinces for TA to be provided by CAP to the SAG for the coming year. Following the workshop, CAP considers the requests from the provinces and uses the following criteria to guide its decision-making: • Does the province have a priority site, selected from the National Priority Sites as decided and communicated by USAID? • Is there provincial and/or district buy-in to the integrated program? • Is the province consistent with coordination of CAP activities? Some provinces have cancelled activities at the last minute due to inadequate coordination and planning. • Is the province/district willing to share the cost with CAP? Through this process, CAP develops its work plans, which are then approved by USAID. Overall, DOH officials were satisfied with the annual strategic planning workshops. There was a shared understanding that the purpose was to ensure that there was a uniform approach and understanding towards infant feeding and PMTCT: “Having everyone together helps to cement the vision” (provincial DOH official). DOH officials expressed the need for more than one day to fully unpack DOH training needs as well as review progress made with trainings the previous year. A suggestion was made that Provinces be invited to the strategic planning and present two key requests each. Officials were not as satisfied 184 South Africa Umbrella Grants Management (UGM) Performance Evaluation with planning follow-up, as CAP was not able to accommodate all training needs. Nonetheless, training commitments made by CAP were kept (with one exception in KZN due to funding challenges experienced by CAP). DOH officials expressed some unhappiness with the timing of the CAP strategic planning workshop. The CAP funding cycle begins in October and as a result the organization has opted to organize the national strategic planning in September. However, the DOH funding cycle begins in April and has its own strategic planning and budgeting processes, which includes planning for capacity building, starting in November/December. Some officials expressed a preference for CAP to align its planning cycle with DOH, rather than the other way around. In addition, some DOH officials would prefer greater input on the training dates that CAP schedules to better fit with provincial planning activities. The final CAP training plan has not been communicated to national DOH; rather each province receives communication of the training that will be provided as well as the proposed dates. This was identified as a gap by the national DOH, which wants to see transparency in the way that CAP decides where to train. Trainings All DOH respondents felt that the quality of CAP training was very good and in line with national and provincial priorities and policy shifts. Provincial officials viewed the CAP training process as a “partnership”: “We have walked a path together over the years…CAP is open to new ideas and policies and willing to adapt training to accommodate these changes” (PDOH official). In one province (Western Cape), CAP trainings were seen as instrumental in bringing different Directorates on board to work together in responding to maternal health in the context of HIV and the importance of exclusive breastfeeding. The trainings are seen by some provinces as having been instrumental in contributing to positive changes in exclusive breastfeeding (EB) practices. According to one respondent, in WC a few years ago up to 98% of PMTCT clients opted to formula feed; this has shifted to around 75% with a growing number of clients opting for EB. This change is evident throughout the province and can be attributed to a large extent to the CAP trainings. National DOH and another NGO partner have also been offering training on this in the province, but not as prominently as CAP. All DOH officials interviewed expressed a need for copies of the various training curricula. Currently they receive the training program outline, Power Point slides and supporting documentation but to date they have not received the learner or facilitator manuals. This would enable them to better assess whether content is current and reflects provincial priorities and to highlight content areas that require special attention or could be left out. CAP reported to the ET that they have not circulated the curriculum because it has not yet been finalized by NDOH. In 2007, Gauteng Province asked CAP to help review the national curriculum; CAP invited CDC and national PMTCT to be participants in this exercise. The review and revisions were completed in 2009 and the curriculum was given by CAP to NDOH. NDOH hired a consultant to further review the curriculum and, five years later, has not yet endorsed it. CAP uses the (proposed) integrated course (both 5- and 10-day versions) but doesn't have the finalized NDOH curriculum. CAP has reportedly given national and provincial DOH this curriculum on CD. CAP also reported that their own financial constraints meant that they could not print the materials for all participants. The Provincial Regional Training Centers don't have the integrated training curriculum 185 South Africa Umbrella Grants Management (UGM) Performance Evaluation either, since it has not yet been formally endorsed. While CAP has given reasons to provinces as to why they have not circulated the training curriculum they use, this explanation does not seem to have satisfied the national and provincial DOH officials who continue to ask for the curriculum. None of the CAP training courses is accredited by the Health and Welfare Sectoral Education and Training Authority (HWSETA) as yet. CAP attempted to get its M&E training course accredited in 2010, but it had been developed for unit accreditation and the SETA was no longer accrediting units. Due to various changes in the SETA landscape it is not now possible to get the curriculum approved. Lack of accreditation is not, however, an issue for DOH. On-site mentoring and coaching at central hubs While CAP activities at facility level are limited (described by a DOH official as a ‘drop in the ocean’), mainly due to capacity and budget constraints, their impact has been significant in key areas. Districts especially value the on-site mentoring support from CAP: “It motivates the District to get their act together if they have someone coming back regularly; it is a motivating presence” (PDOH official). Examples of visible improvements at two hub sites are as follows: • In KZN, visible improvements in the 18 month testing rate are evident in the one facility where CAP is working. The value added by CAP was attributed to the hands-on translation of theory into practice: “When you go to the site they don’t just look at the overall numbers, are women and babies being tested; now we are looking together at the source document (the registers) because when you look at the registers you see what is really happening. For example, if we have 100 babies and 40% prevalence we need to make sure that these 40% are coming for their immunizations and the only way we make sure is by checking the registers” (District DOH official). • Tangible improvements have been observed in Gugulethu in WC, where CAP has provided mentoring support since 2008. The general consensus seems to be that “somehow things are going better at this site” (PDOH official). A big increase in mothers from the Gugulethu center who are exclusively breastfeeding has been noted by doctors. In addition, there have been: an increase in pregnant women booking early for ANC, fewer mothers arriving for delivery who haven’t been HIV tested, and more HIV positive mothers initiated on ART. Community level follow up of mothers after discharge has been strengthened so fewer mothers fail to return for PNC visits or ART. The mother to child infection rate at Gugulethu is 0.8% (compared to overall rate of 3.2% in the province – 200?? figures). “Gugulethu has seen a remarkable turn-around in PMTCT program feeding choice at discharge since 2010, from almost all formula fed babies to mostly breastfed babies. This is borne out by the PMTCT statistics. I’m given to understand that this is largely the result of the CAP program, which I wish could be replicated in other high burden areas” (OB/GYN doctor, WC). One of the unintended consequences of improvements in hub sites observed by CAP was that when the District recognized hub sites that had been trained, they got flooded with more patients and then didn’t’ perform as well. As a result, improvements tended to be more sustained in the other sites in the sub-district than at the hub sites. PDOH officials reported that they had not observed this “cascading” effect and had only observed improvements at the hub site. As a result they were very keen for CAP to expand their coaching and mentoring support to other sites. For all the sites where CAP provides on-site CB support, DOH recognizes their contribution but commented that it is difficult to attribute all the success to CAP alone. An impact evaluation of CAP’s work at sites is needed to provide evidence to provinces that this model is the one that should be replicated across all sites. 186 South Africa Umbrella Grants Management (UGM) Performance Evaluation Technical assistance to national DOH CAP has provided TA to NDOH by leading in the development of an integrated PMTCT curriculum, toolkits and guidelines. While CAP played a lead role, NDOH was clear that the development of this curriculum was done under its direction and with the support of other service providers. CAP also sits on PMTCT TWG, has contributed to updating the nationally approved health curricula and reviewing national training, and also provided assistance for the development of a national DOH nutrition strategy. CAP is well-respected by national and provincial DOH as a provider of quality training aligned with international and national and provincial policies and priorities. It is one of a handful of NGOs providing CB support to DOH in integrated PMTCT, but the only organization with a specific focus on nutrition and the only organization providing training for Baby Friendly Hospital Initiative assessors in South Africa. While DOH values the contributions of CAP they would not want an exclusive relationship with them as the only specialist TA support provider. In addition, NDOH expressed a desire to be more involved in shaping the “research agenda” pertaining to the integration of PMTCT with nutrition and maternal health. There was a sense that currently government does not have an adequate role shaping this agenda, and that universities and outside donors may be perceived as playing an overly strong role. A further challenge for NDOH was having multiple donors/service providers working in the same area with competing agendas. The need for alignment and clearly demarcating areas of work was identified. It appears that CAP has the necessary in-house technical capacity to respond to emerging developments in the field of integrated PMTCT with nutrition and maternal health, such as family planning and male involvement in PMTCT. This is important for them to remain relevant as a specialized technical service provider. FHI360 has initiated male involvement in a PMTCT project in Zambia, and CAP is in the process of trying to adapt the process in South Africa through CB of community health workers in some target sites. Where feasible, CAP Technical Advisors and Program Manager attend technical updates in PMTCT and Nutrition in country; the team is part of the professional e-mail groups in the mailing list of WHO and UNICEF and other International organizations working in the field. In addition Home Office Global Population Health and Nutrition staff regularly update CAP technical staff on current issues. DOH PMTCT and Nutrition Directorates invite CAP to attend workshops and also international updates if conducted in South Africa. Minimal external TA has been provided by CAP, for example, to support the costs of a regional health official to participate in the Breastfeeding Summit in South Africa. Challenges in the area of CB Communication and Coordination The previous section highlighted challenges related to specific CB activities, whereas this section focuses specifically on challenges relating to communication and coordination. One of CAP’s main challenges is that they do not effectively communicate the impact of their work to national or provincial DOH and other stakeholders. For example, CAP reported that health care workers at the WC and NC hub sites had successfully put together poster presentations for a 187 South Africa Umbrella Grants Management (UGM) Performance Evaluation conference hosted by the Public Health Association of SA. However, this achievement was not mentioned at all by DOH national or provincial officials. Another key challenge is that CAP attempts to integrate national policies and guidelines from two key DOH Directorates (PMTCT and Nutrition) as well as the HIV/AIDS Directorate into seamless program services. In WC and KZN, the management of Nutrition, PMTCT and MCWH functions are integrated, which facilitates CAP’s communication in these provinces to some extent. At the NDOH, however, these functions are not yet integrated and CAP’s communication with the relevant officials is not as effective as it should be. There is visible evidence of the positive contribution of CAP’s on-site coaching and mentoring but this is only fully acknowledged by staff at the district/sub-district level and to some extent by provincial staff who work directly with the program. CAP’s contribution in integrating nutrition and PMTCT is not receiving the recognition that a program of this caliber deserves. Overall, the Nutrition Directorate was satisfied with the level of communication with CAP; the other Directorates, PMTCT and HIV Prevention, were less satisfied. In WC, from the perspective of the Nutrition Directorate, the integrated nature of the CAP training contributed to opening lines of communication and a more integrated approach to service delivery between the Nutrition, PMTCT and HIV Prevention Directorates. CAP tends to report to national and provincial DOH on request rather than through a formal reporting system. In WC, KZN and NC, CAP has been invited to quarterly steering committee meetings where the organization is required to report on activities and attendance numbers. A more formal reporting system is needed at national level. CAP has signed MOUs with four provinces - KZN, NW, NC and WC. The MOUs help to establish terms of reference for CAP and serve as an approval document for TA in the provinces. Obtaining this approval in some provinces has been a challenge in as the DOH approval system is slow and not always clear. It is however not a requirement for CAP to have a signed MOU with the province to be able to provide CB support. If they were implementing/providing health services, then an MOU would be essential. No formal MOU has been signed with the NDOH, however USAID has been informed through telephone consultation and quarterly reporting about TA provided at national level. Recommendations The findings of this rapid review show that DOH considers CAP a valuable CB partner for an integrated PMTCT response in South Africa. Some recommendations below focus on how to strengthen the provision of USAID funded specialist TA to DOH, and others focus on how to position and strengthen the role/contribution of CAP. Consider a more coordinated approach to integrated PMTCT CB with all relevant partners USAID support an annual integrated PMTCT CB strategic planning process: The CAP annual strategic planning, even with its limitations, has been identified as a positive practice for DOH. Officials appreciate an NGO taking the time to consult with them collectively about CB needs. This strategic planning process would be substantially strengthened if it were led by DOH and included the other specialist service providers in the field of integrated PMTCT CB (e.g. Mothers to Mothers, 188 South Africa Umbrella Grants Management (UGM) Performance Evaluation South to South, Desmond Tutu Foundation, and PATH). Each service provider could be given an opportunity to present its CB models (most of which appear to include a site-level health systems strengthening element), following which national and provincial DOH could present their CB needs and identification and allocation of priority areas to the different service providers. A strategic planning process of this nature should be led by national DOH, who could be supported by an external service provider funded by USAID. This approach would assist DOH to streamline PMTCT CB in the provinces, avoid duplication and maximize available resources and training materials. To illustrate, in Gugulethu the Desmond Tutu Foundation has done trainings for PMTCT that incorporate ANC registers and focus on M&E data. CAP is very good at the breastfeeding component. The hub health facility would like to see a training rolled out to all the districts that combines both these elements. Strengthen communication about the CAP program and reporting to DOH CAP needs a communication strategy that will enable them to effectively communicate the nature and impact of their CB model to stakeholders, particularly DOH. The strategy should include structured reporting processes to DOH at a national and provincial level as well as simple, clear communication materials. Conduct impact evaluations to provide evidence for wider replication of the CAP model National and provincial DOH require research-based/empirical evidence about how the CAP CB model is contributing to the positive outcomes that have been observed in most of their central hubs and some surrounding sites. Baseline studies have been undertaken in most of the sites and follow-up impact evaluations are scheduled for 2012. CAP needs to ensure that these findings are clearly and effectively communicated to relevant national and provincial DOH officials to assist them with decision-making about whether to replicate the model nationally. Scale-up TA for the training of community health workers as part of the Primary Health Care Re-engineering strategy An opportunity exists for CAP to move into training of community caregivers linked to the Primary Health Care Re-engineering process currently underway. CAP has developed expertise and an effective methodology for building the capacity of both health workers and community caregivers and so is well-positioned to fill this training gap. 189 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX H. UMBRELLA GRANT MECHANISM (UGM) SUMMARIES 1. FHI360 2. PACT 3. RIGHT TO CARE (RTC) 190 South Africa Umbrella Grants Management (UGM) Performance Evaluation FHI 360 In October 2007, USAID awarded a Cooperative Agreement of approximately USD $91 million to the Academy for Educational Development (AED) to implement the Umbrella Grants Management (UGM) Program in South Africa until September 30, 2012, under Cooperative Agreement 674-A-00- 08-00002-00. AED has been managing the UGM Program within the South African Government (SAG) framework of scaling up HIV/AIDS program implementation and coordination at local, provincial, and national levels. The AED-UGM program aims to achieve the following results over the life of the program: (i) Increased capacity of South African organizations to implement USAID funded programs; (ii) Increased sustainability of USAID’s HIV prevention, care and treatment efforts and enhanced country ownership of USAID funded projects; (iii) Increased number of indigenous organizations moving from sub-funding under USG-based partners to direct USG funding with no intermediary; and (iv) Improved networking among USAID partners. Initially, six organizations were included in the program from 2007: Anglican AIDS and Healthcare Trust (AAHT) Hospice Palliative Care Association of South Africa (HPCA) – now graduated Ingwavuma Orphan Care (IOC) Population Council (Pop Council) – now graduated Woz’obona Early Childhood Community Service Group (Wozo) Senzakwenzeke (Senza). In 2008, two partners were added: Greater Nelspruit Rape Intervention Project (GRIP) Project Concern International (PCI). In 2009, 3 new partners were seconded to AED-UGM by USAID in South Africa, bringing the current total number of partner organizations to 11. The new grantees include: • Hands at Work in Africa (HAW) • Heartbeat (HB) • Humana People to People South Africa (HPP-SA) • Nurturing Orphans and AIDS for Humanity (NOAH) • Medical Care Development International (MDCI) Table 1. FHI 360 Sub-Partners (2011-2012) Source: FHI 360 Annual Report, 2011 PARTNER HQ PROGRAM AREA 1. Anglican Church of Southern Africa Western Cape OVC 2. Ingwavuma Orphan Care KZN OVC, TB, VCT 4. Wozobona/ Sekhukhune Education Project Limpopo OVC 5. GRIP Mpumalanga HBHC,VCT 6. Project Concern International Western Cape Prevention 7. Hands at Work Mpumalanga HBHC, OVC 8. Heartbeat Gauteng OVC 9. Noah Gauteng/ KZN OVC 10. Humana People to People Gauteng/ KZN Prevention, HBHC, VCT 191 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 2. FHI 360 Budget Allocations to Sub-Partners Source: FHI 360 Annual Report, 2011 On June 30, 2011, Family Health International (“FHI 360”) acquired substantially all of AED’s assets—including its programs and valuable technical staff. This acquisition enabled the UGM team in South Africa to successfully continue implementation activities during the reporting period. To attain these objectives, the key strategies employed by the UGM project across all programmatic, technical and organizational domains include: 1. Core, cohort and customized training 2. Educational Training Fund (ETF) 3. Technical assistance (TA) 4. Mentoring and twinning Given the strategic decision to assist partners in achieving organizational sustainability the UGM project prioritized the use of consultants for capacity building in specialized organizational development domains. These specific domains include Strategic Management/Governance, Organizational Management/Human Resources and Program Sustainability. 192 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 3. Capacity Building and M&E Focus Area M&E FOCUS AREAS CAPACITY BUILDING FOCUS AREAS • MER plans • Data collection (tools and processes for monitoring) • Data quality verification plan • Data analysis and review of UGM and grantee level data • Reporting • Governance and Strategic Planning • Organizational Management an • Human Resources • Project Planning and Design • Financial Planning and Management • Grants Management • Monitoring and Evaluation • Technical Capacity Mentoring and Coaching • Networking and Advocacy • Program Sustainability Table 4. SP Organizational Assessment Scores (2007-2011) DOMAINS AVERAGE SCORES (2008) AVERAGE SCORES (2010-2011) AVERAGE IMPROVEMENT Governance and Strategic Planning 75 83 8% Mentoring and Coaching 39 60 21% Project Planning and Design 80 90 10% Financial Planning and Management 71 84 13% Organizational Management and Human Resources 67 85 18% Grants Management 50 69 19% Technical Capacity 71 88 17% Monitoring and Evaluation 85 85 4% Networking and Advocacy 91 95 4% Program Sustainability 50 50 36% Source: FHI 360 Annual Report, 2011 193 South Africa Umbrella Grants Management (UGM) Performance Evaluation Figure 1. FHI 360 Organizational Chart Source: FHI 360 Annual Report, 2011 194 South Africa Umbrella Grants Management (UGM) Performance Evaluation PACT Pact SA began its UGM arrangement with USAID/SA in 2004 and was initially the only UGM partner, through the centrally funded Community REACH program. This first agreement contained no capacity building component, but was intended solely to provide grants management for USD72 Million over a 5-year funding period. Pact chose to provide capacity development as part of its grant making support to SP out of its organizational philosophy. After the third year, a determination was made that the UGM fund was too large to be managed by a single UGMP; the UGM mechanism was then competed by USAID/SA, and two additional UGMP were selected along with Pact. Since 2007, Pact has managed USD239 million over 5 years. At the time of this evaluation, Pact was managing a combined total 46 SP and SPP under its UGM mechanism, 13 of which are South African organizations. SP partners at the time of the 2012 annual report are listed below (Olive Leaf Foundation is no longer an SP). 195 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 1. Pact Sub Partners (2011-2012) PARTNER GEOGRAPHIC PRESENCE PEPFAR PROGRAM AREA Africa Centre for Health and Population Studies KZN PMTCT, HIV Care and Support – Adult and Pediatric, HIV Treatment – Adult and Pediatric, HIV/TB Care and Support, HIV Counseling and Testing CARE EC, FS, KZN, LP OVC Care and Support Child Welfare South Africa (CWSA) NC, EC, SF, KZN, GP, NW, MP, LP, WC OVC Care and Support Childline Mpumalanga (CLMP MPU OVC Care and Support Childine South Africa (CLSA) LP, NW, GP, EC, KZN, FS OVC Care and Support Children in Distress Network (CINDI KZN OVC Care and Support CompreCare Joint Venture GP, NW, MPU OVC Care and Support Health and Development Africa (HAD FS OVC Care and Support Kheth’Impilo EC, KZN, MPU, WC PMTCT, HIV Care and Support, HIV Treatment – Adult and Pediatric, HIV Counseling and Testing Living Hope WC HIV and TB Care and Support, HIV Counseling and Testing mothers2mothers (m2m) EC, GP, MPU, WC PMTCT Mpilonhle KZN OVC Care and Support, HIV Prevention, HIV Counseling and Testing National Association of Child Care Workers (NACCW) WC, FS, NW, LP, KZN OVC Care and Support Olive Leaf Foundation (OLF) EC, GP, KZN, LP, WC HIV Prevention, HIV Counseling and Testing, HIV Care and Support, OVC Care and Support Save the Children (SC FS, LP OVC Care and Support Starfish Greathearts Foundation EC OVC Care and Support World Vision EC, FS, LM OVC Care and Support, HIV/TB Care and Support Capacity Building Pact focuses on five key areas: Finance and grants management, M&E, Program, Organizational Development and Governance. Its training series for SP includes basic grants management training that focuses on compliance with USAID rules and regulations, minimum financial management standards, and principles of good management and governance. Pact also offers courses on monitoring, evaluation, reporting and learning (MERL), data quality assurance, and basic program evaluation. SP have opportunities to attend courses on resource mobilization, volunteer management, and board leadership and governance. Refresher courses are offered to address issues 196 South Africa Umbrella Grants Management (UGM) Performance Evaluation of staff turnover within SP organizations. Pact complements workshops with ongoing mentoring to ensure that knowledge gained in trainings is put into practice. Pact assists each SP to develop an M&E system for tracking progress against both PEPFAR and organizational indicators. Pact’s MERL technical assistance to SP includes review and development of effective data collection tools, analysis and reporting of data, setting realistic targets, establishing and strengthening data quality management systems, conducting internal data quality audits, and verifying/validating data submissions. Pact provides TA to SP through formal assessments as well as routine program monitoring and site visits that result in recommendations for strengthening specific program aspects. Coaching is provided by phone and email. Pact also contracts external consultants to provide targeted technical support for SP on a case by case basis. Pact’s OVC sub‐contracts portfolio is utilized to procure and manage external technical support for the South African Government (SAG) Departments of Social Development and Department of Health. This component of Pact’s program primarily responds to the needs and priorities of the SAG and partners as communicated by USAID. Contracted support targets national level OVC policy development and related programming. Funds in this line item are also utilized to support cross‐ cutting PEPFAR OVC activities. Financial and Grant Management Pact conducts a participatory organizational capacity assessment (OCA) of each partner organization at beginning and midpoint of the grant award. OCA focuses on the organization’s governance, management capacity, staffing and strategic development. Pact and partner organization use the results to develop partner support plans (PSP) that outline interventions to strengthen the organization. At beginning of each FY, Pact supports SP to develop detailed annual work plans and budgets. Following work plan and budget approval, maintains frequent contact with partners, monitoring program progress in order to facilitate early identification and resolution of implementation issues. All partners submit monthly financial reports and supporting documentation. Pact finance staff visit partners regularly to assist in strengthening financial management systems, audit program expenses and accompanying documentation, monitor compliance with USAID rules and regulations, and review progress towards meeting cost share requirements 197 South Africa Umbrella Grants Management (UGM) Performance Evaluation Table 2. Pact Budget Allocations to Sub-Partners PARTNER CURRENT FY ALLOCATION TOTAL ALLOCATION ALL YEARS BALANCE OF ALLOCATION Africa Centre for Health and Population Studies $4,125,054 $16,563,353 $205,434 CARE $3,010,035 $7,444,570 $246,320 Child Welfare South Africa (CWSA) $1,812, 281 $6,909,327 $185,923 Childline Mpumalanga (CLMP $1,285,000 $2,812,252 $ 714,617 Childine South Africa (CLSA) $750,000 $ 1,125,000 $148,338 Children in Distress Network (CINDI $ 970,905 $3,616, 810 $110,025 CompreCare Joint Venture $2,971,552 $6,831,552 $1,285,183 Health and Development Africa (HAD $645,651 $2,025,284 $22,842 Kheth’Impilo $11,679,991 $25,261,199 $1,645,709 Living Hope $390,235 $2,230,470 $37,917 mothers2mothers (m2m) $7,561,432 $24,464,311 $954,526 Mpilonhle $1,234,020 $2,520,469 $138,099 National Association of Child Care Workers (NACCW) $4,688,007 $19,030,484 $1,347,117 Olive Leaf Foundation (OLF) $2,943,974 $14,169,103 $361,134 Save the Children (SC $3,296,221 $11,837,442 $433,134 Starfish Greathearts Foundation $776,724 $1,747,629 $511,815 World Vision $4,117,538 $14,508,753 $0 Source: Pact Annual Report, 2012 198 South Africa Umbrella Grants Management (UGM) Performance Evaluation RIGHT TO CARE Right to Care (RTC) Summary Right to Care (RTC) is a South African NGO, that uses USAID PEPFAR funding under an umbrella grant mechanism (UGM) award to capacitate three NGO providers of HIV-related health care services: South to South, Agri-Aids, and Lifeline. RTC also received an anti-retroviral treatment (ART) award under the same RFA, covering a large number of other NGOs. RTC was established in 2001 and has expanded with the awarding of PEPFAR funding to the organization by USAID. With funding from USAID/PEPFAR, RTC has, since 2009, helped a total of 23 partners build their institutional capacity and organizational infrastructure to support and implement HIV and TB prevention, care, and treatment models. Relevant RTC strategic drivers for the UGM award are to: • To maintain effective, paper-based as well as electronic data management systems to track and disseminate key data about RTC programs • Improve organizational planning and management to enhance accountability. Table 1. Right to Care UGM Sub-Partners 2010-201 RIGHT TO CARE UGM PARTNERS 2010 RIGHT TO CARE UGM PARTNERS 2011 • Agri-Aids • Gold • Lifeline • South to South • Agri-Aids • Lifeline • South to South RTC states that the overall goal of the UGM is to support high quality, sustainable services in HIV/AIDS. A focus for RTC is to ensure that policies, guidelines and processes are in place to measure and evaluate by providing support to UGM partners in the following areas: • Program management • Monitoring and evaluation • Financial administration and management • Organizational development UGM current partner summaries follow: 1. AgriAids facilitates, organizes, implements and monitors HIV/AIDS prevention and HIV counseling and testing (HCT) programs in farming areas in Limpopo (LP), North West (NW), KwaZulu Natal (KZN) and Mpumalanga (MP). 2. LifeLine has a HCT program which uses mobile HCT units deployed in hotspots including farming and mining areas in and around the Bojanala District in the North West. 199 South Africa Umbrella Grants Management (UGM) Performance Evaluation 3. South to South (S2S) a specialist in the Prevention of mother-to-child HIV transmission (PMTCT), Pediatrics and Psychosocial programming), implements a Comprehensive Family HIV Care and Treatment, and responds to clinical and health system strengthen needs in the Bojanala and Tshwane Districts. RTC introduced a new Quarterly Review Process during COP10. It focuses on Finance, Programs, Compliance, and M&E. It is conducted by a RTC NGO/UGM team which includes a representative from each of the above sub-departments. The review is conducted in phases: • Phase 1-Internal review by RTC team • Phase 2-Feedback meeting between UGM partners and RTC • Phase 3-Action plans drafted based on feedback • Phase 4-Implementation of action plans and follow up The review-reporting tool is used to document findings, action plan and follow up on action. In addition, RTC launched a web-based; management and reporting system called “RightMax.” Table 2. Right to Care Budgets by Country Operating Plan (COP) Year (In $ US) PARTNERS COP08 COP09 COP10 COP11 TOTAL SAIHCM $100,765.00 $0.00 $0.00 $0.00 $100,765.00 Gold $0.00 $407,780.00 $448,494.29 $0.00 $856,274.29 AgriAids $93,415.00 $513,996.86 $638,557.00 $700,000.00 $1,945,968.86 Life Line $0.00 $534,205.43 $411,517.57 $414,285.71 $1,360,008.71 S2S $0.00 $0.00 $3,535,695.00 $2,900,000.00 $6,435,695.00 $10,698,711.86 200 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX I. OPTIONS FOR FUTURE UGMS USAID has requested that the evaluation team consider various options for future UGM configurations based on current experience, and indicate possible advantages and disadvantages of each UGM option. These are presented briefly below in pairs to compare and contrast what each type of UGM might offer. The pairs are not necessarily mutually exclusive. A UGM could incorporate elements of any or all pairs. The final decisions will belong to USAID. ONE UGM VS. MULTIPLE UGMS The advantages of multiple UGMs include the power of competition, choice/alternatives among UGMs, and fail-safe options if one or another UGM under-performs, or a particular partner-UGM combination proves to be unworkable. The advantages of one large UGM, as suggested by the Pact example, include a consolidated management structure, fewer management units for USAID, and possibly more centralized expertise under one roof – a one-stop shop. The disadvantage is no option if the UGM fails. TECHNICAL EXPERTISE UGM VS. ORGANIZATIONAL DEVELOPMENT UGM There are advantages of grouping sub-partners under a UGM by their PEPFAR programming areas, for example, clinical partners (ART, PMTCT, MMC) and non-clinical partners (Prevention, OVC, Home-Based Care, and Counseling and Testing). This would allow the UGM to staff up in the relevant areas of expertise to provide TA to partners. However, not all partners fall neatly into clinical vs. non-clinical categories, and current experience suggests that even if they do, their capacity building needs may be very different. To group partners by OD levels and focus on CB needs would suggest a UGM which is strong in CB but might not attend as well to partners’ needs for TA. This might result in them growing stronger as NGOs, but not improving significantly in terms of quality of services, thus helping the NGO, but not the clients it serves. GEOGRAPHIC UGM VS. FUNDING LEVEL UGM Another suggestion is to seek UGMs to handle geographic areas. For example, USAID might suggest dividing the country into two-three zones, and solicit UGMs to manage all the partners in each zone. This would address sub-partner requests for more on-site TA and CB, by allowing UGMs to position more staff in the field, closer to the sub-partners. However, not all partners fall neatly into any particular geographic grouping. How would USAID and UGMs handle partners that operate in two or more zones? Another grouping could put the largest sub-partners with one UGM, and small-to-medium-sized partners with another. The largest sub-partners would presumably be targeted to become primes on their own within a specified period of time, focusing on diversifying resources, and weaning off of 201 South Africa Umbrella Grants Management (UGM) Performance Evaluation reliance on USAID funding progressively as they develop other revenue streams. The small-medium NGOs would presumably need more CB to grow as organizations. Their UGM would allocate more funding per NGO and require more staff and/or consultant services to facilitate their development. INDIGENOUS VS. INTERNATIONAL UGM Competition-driven UGM models A further option is for USAID to outline some or all of the options above, indicating the kinds of concerns it wants UGMs to address. USAID could indicate that it would fund one or more UGMs, and invite interested applicants to propose one or more models in reply. This might be the most attractive to USAID, as it would put the burden on applicants to develop new models, form consortium groups, and use other strategies to address the many concerns USAID faces. This might result in larger or indigenous INGOs grouping together, and forming partnerships that also include smaller NGOs which are themselves umbrella organizations. These partnerships might address how to deal with geographic dispersion of NGOs, technical assistance needs, large and small NGOs, and so on. They might also propose to work with interesting groups or combinations, such as: • Rising Stars: One or more indigenous NGOs which are on the verge of becoming “prime partners” and only require a limited amount of support to stand on their own • Geographic or Affinity Clusters: New approaches to assist clusters of NGOs based on geographical groupings, programmatic or technical needs, or “communities of practice” which learn together • Emerging Partners: New or small NGOs which will require significant CB to grow and to develop, but whose programming or geographic location (i.e., remote areas) attract USAID • Local vs. International Groupings: Approaches that might pair local NGOs with international NGOs, or suggest a variety of approaches that might be best-suited to the different needs of these NGOs. • Sustainability-Oriented Models -Government Alignment - Public-Private Partnerships: This could include approaches designed to increase the degree of sustainability of NGOs measurably on one or more dimensions, for example, by aligning them more closely with government strategies, or enabling them to develop public-private partnerships, possibly leading to “spin-off” organizations which could provide a marketable service or a sell product to raise revenue and turn a profit. Also for consideration could be the establishment of endowments or other long-term funding options 202 South Africa Umbrella Grants Management (UGM) Performance Evaluation APPENDIX K. REFERENCES AED, FHI 360. Annual and Quarterly Reports. 2007-2011 AED, FHI 360. Annual Work Plans. 2007-2011 CAP South Africa. Integrated PMTCT Program Model. March 2012. CAP South Africa. Integration of PMTCT, Maternal Health and IYCF into Health Facilities and Community Services: Course and Activity Descriptions. 2012. CAP South Africa. Strategic Plan October 1, 2011-September 30, 2012. September 30, 2011. CAP South Africa. Support for Integration of PMTCT, Maternal Health and Infant and Young Child Feeding into Health Facilities and Community Services. CASE, Children’s Institute. PPT Presentation: Government and Donor Funding of Children’s Act￾related services, Key Findings. Department of Health. PHC re-engineering in South Africa: are we making progress?. Department of Health. PPT Presentation: PHC Re-Engineering. Department of Health. Provincial Guidelines for the Implementation of the three Streams of PHC Re-Engineering. September 2011. Department of Health. The implementation of PHC re-engineering in South Africa. Department of Social Development. Chief Directorate HIV AIDS. Department of Social Development. Children Court Acts. Department of Social Development. Children's Acts. Department of Social Development, DIFD. Management Capacity Situational Analysis and Needs Assessment of Home and Community Based Care Service Providers: HIV&AIDS Multi-Sectoral Support Program. January 2006. Department of Social Development. M & E Framework. Department of Social Development. Norms & Standard. Department of Social Development. NPF Final. Department of Social Development. Practice Guidelines for Intercountry adoption. Department of Social Development. Practice Notes for children's Acts. Department of Social Development. Training Guidelines. DFID Human Development Resource Centre. Overview of Health Sector Reforms in South Africa. December 2011. Elizabeth Glaser Pediatric AIDS Foundation, ICF. Implementation Guide for the Organizational Capacity and Viability Assessment Tool. 2011. 203 South Africa Umbrella Grants Management (UGM) Performance Evaluation Elizabeth Glaser Pediatric AIDS Foundation, ICF. The Organizational Capacity and Viability Assessment Tool. February 2012. Facilitator’s Guide: Mother Nutrition, Infant and Young Child Feeding in the context of PMTCT. 2009. Family Health International, Capable Partners. Annual and Quarterly Reports. 2007-2011. Family Health International, Capable Partners. Country Operational Plans. Family Health International, Capable Partners. Letters of Authority. Family Health International, Capable Partners. LINKAGES FINAL REPORT. Family Health International, Capable Partners. Memorandums of Understanding (MOUs). Family Health International, Capable Partners. Proposals for RAP in Provinces. Family Health International, Capable Partners. RAP Reports. Family Health International, Capable Partners. Sub Partner Documents. Governance and Strategic Planning Training. September 2008 The Government of the Republic of South Africa and The Government of the United States of America. Partnership Framework in Support of South Africa’s National HIV & AIDS and TB Response. 2012/13-2016/17. Learner’s Guide: Mother Nutrition, Infant and Young Child Feeding in the context of PMTCT. 2011. LTI Strategies, Keystone Accountability. Pact South Africa HIV/AIDS Grant Management Program Evaluation Report. March 2010. National Treasury, Republic of South Africa. PPT Presentation: Budget 101: Describing the national and provincial budget cycle that determines budgets for the provincial departments of social development. May 25, 2011. National Treasury, Republic of South Africa. PPT Presentation: Provincial Budget Process. Pact. Introduction to Organizational Capacity Development, Pact Organizational Development Toolkit-First Edition. January 2010. Pact. Mentoring and Coaching Module. November 2010. Pact. Training: Basic Management Skills. 2007-2011. Pact. Training: Board Training or Induction. 2007-2011. Pact. Training: Care Worker Management. 2007-2011. Pact. Training: Cross Cutting Needs. 2007-2011. Pact. Training: Pact. Training: Mentoring as Capacity Building Approach. 2007-2011. Pact. Training: Organizational Capacity Assessment. 2007-2011. Pact. Training: Organizational Governance and Leadership. 2007-2011. 204 South Africa Umbrella Grants Management (UGM) Performance Evaluation Pact. Training: Proposal Writing. 2007-2011. Pact. Training: Resource Mobilization. 2007-2011. Pact. Training: Strategic Planning. Pact South Africa. Budget Guidelines. 2010 Pact South Africa. FY 2011 Semi-Annual Progress Report, HIV/AIDS Umbrella Grants Management Program. October 2010-March 2011. Pact South Africa. PPT Presentation: Board Orientation Training. July 2011. Pact South Africa. PPT Presentation: Comprecare Partners Training in NPO Governance. October 5-7, 2010. Pact South Africa. PPT Presentation: Conducting a Successful Strategic Planning Process. March 8-10, 2011. Pact South Africa. PPT Presentation: Training in Basic Management Skills. January 11-12, 2012. Pact South Africa. PPT Presentation: Resource Mobilization Workshop for AMREF’s Sub Partners. June 2010. Pact. Whistleblower Policy. PEPFAR South Africa. COP Report. 2010 Right to Care. COP11 UGM Work Plan. November 2011. South African Department of Social Development, 4Chakras Consulting. Norms, Standards and Practice Guidelines for the Children’s Act. May 2010. South African Department of Social Development. PPT Presentation: Review of the Policy on Financial Awards to Service Providers. May 24, 2011. Southern Hemisphere Consultants. Integrated Report on the Summative Evaluation of the Management Capacity Building & Mentorship Programme of Home Community Based Care (HCBC) Organisations, Mentoring Organisations & District Officials. November 2008. Umbrella Grants Management. Aligning OVC Partners Programs to National and PEPFAR Guidelines Workshop, Participant’s Handbook, May 4-5, 2010. Umbrella Grants Management. Partner Governance Improving Governance with Score Card and Succession Planning Workshop, Participant’s Handbook. December 2010 Umbrella Grants Management. PPT Presentation: Partner Kick off Meeting, October 4, 2010. UNAIDS. Getting to Zero: 2011-2015 Strategy. 2010. United States Agency for International Development. Annual Program Statement in Support of the US President’s Emergency Plan for AIDS Relief in South Africa, APS 674-07-001. United States Agency for International Development. Building Local Development Leadership: USAID’s Operational and Procurement Improvement Plan. United States Agency for International Development. Proposals and Work Plan 2009-2011 for all UGM Partners. 2009-2011. 205 South Africa Umbrella Grants Management (UGM) Performance Evaluation The U.S. President’s Emergency Plan for AIDS Relief. Fiscal Year 2007: PEPFAR Operational Plan. The U.S. President’s Emergency Plan for AIDS Relief. Fiscal Year 2008: PEPFAR Operational Plan. The U.S. President’s Emergency Plan for AIDS Relief. Fiscal Year 2009: PEPFAR Operational Plan. The U.S. President’s Emergency Plan for AIDS Relief. Fiscal Year 2010: PEPFAR Operational Plan. The U.S. President’s Emergency Plan for AIDS Relief. Fiscal Year 2011: PEPFAR Operational Plan. South Africa Umbrella Grants Management (UGM) Performance Evaluation 206 For more information, please visit http://www.socialimpact.com Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Suite 300 Tel: (703) 465-1884 Fax: (703) 465-1888 www.socialimpact.com