PACT Community Reach Program Evaluation Draft Report By Salephera Consulting Ltd Page 1 Pact Malawi Community REACH Program Final Evaluation Report Associate Cooperative Agreement No.: 690-A-00-07-00004-00 21st March 2012 This document was prepared for the United States Agency for International Development (USAID) by Salephera Consulting Ltd, Private Bag 152, Lilongwe, Malawi. E-mail: hesternb@gmail.com Tel: +265 (0) 888828740 Pact Community REACH Program Evaluation Final Report By Salephera Consulting Ltd Page 2 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. “Thanks to the American People and Their Malawi Partners for making a difference in our communities and lives” OVCs gathered for psychosocial support, near a garden for their food security at MAICC in Dowa District. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 3 Table of Contents List Of Figures ......................................................................................................................................................4 List Of Acronyms And Abbreviations.................................................................................................................... 5 Acknowledgements ................................................................................................................................................ 7 Executive Summary................................................................................................................................................ 8 Section One ...................................................................................................................................................12 1.0 Introduction and Background.........................................................................................................12 1.1 HIV/AIDS Situation in Malawi.........................................................................................................12 1.2 United States of America Government (USG) HIV/AIDS Strategy.............................................14 Section Two ....................................................................................................................................................17 2.0 Purpose, Methodology and Strategy of the Evaluation .............................................................17 2.1 Evaluation Methods and Strategies...............................................................................................17 Section Three .......................................................................................................................................................20 3.0 Findings on PACT Community REACH Program Design ..............................................................20 3.1 PACT Malawi Community REACH Program Contributions.........................................................22 3.2 PACT’s Approach and Achievements in Malawi ..........................................................................22 3.3 PACT Malawi Community REACH Program Best Approaches....................................................24 Section Four ........................................................................................................................................... ..........25 4.0 Findings on PACT Community REACH Program Capacity Building............................................25 Section Five ....................................................................................................................................................40 5. Findings on PACT Community REACH Program Technical Support Interventions......................40 Section Six ....................................................................................................................................................52 6. Findings on PACT Community REACH Program Grants Management and Administration.......52 Section Seven ....................................................................................................................................................56 7. PACT Malawi Community REACH Program Gender Integration and Considerations ................56 Section Eight ....................................................................................................................................................59 8.0 Sustainability.....................................................................................................................................59 Section Nine ....................................................................................................................................................60 9.0 Conclusions and Recommendations .............................................................................................60 ANNEX ANNEX 1: Reference Documents.................................................................................................................62 ANNEX 2: Evaluation Schedule ...................................................................................................................63 ANNEX 3: PACT Evaluation Focus Group Discussion Guide With Communities .........................................64 ANNEX 4: PACT Program Evaluation Capacity Development Questionnaire For Sub-Partners ................67 ANNEX 5: PACT Program Evaluation Questionnaire For PACT Management And Staff..........................71 ANNEX 6: PACT Sub-Partners Stakeholders Questionnaire .......................................................................73 ANNEX 7: Technical Assistance Questionnaire To PACT Sub-Partners.....................................................74 ANNEX 8: Purpose And Scope Of The Evaluation ......................................................................................75 Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 4 LIST OF TABLES Table 1 : PACT Sampled Community REACH program sub-partners: ...............................................18 Table 2: Comparison of Sub-Partner Organizational Development Issues...........................................36 Table 3: Impact of PACT Malawi‘s PMTCT intervention ...................................................................44 Table 4: PACT Malawi system strengthening and Sub-grantee system strengthening ........................55 LIST OF FIGURES Figure 1: Map of Pact partners Geographic Location and services delivered...........................................21 Figure 2: Sub-partners indicating improvements per capacity area......................................................27 Figure 3: Average OCA and Re-OCA Scores ......................................................................................28 Figure 4: OCA and Re-OCA results for strategic planning per partner................................................29 Figure 5: MCAT Scores of Selected Partners.......................................................................................30 Figure 6: Material and Financial Resources OCA and Re-OCA Scores per Partner............................31 Figure 7: External Partnerships and Networking OCA and Re-OCA Scores per Partner ....................33 Figure 8: Risk Levels of Local Partners by Year..................................................................................35 Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 5 List of acronyms and abbreviations ANC Antenatal Care APCA African Palliative Care Association APS Annual Program Statement ART Antiretroviral Therapy AWP Annual Work Plan CABUNGO Capacity Building NGO CBCCs Community Based Care Centres CBDA Community Based Distribution Agents CBO Community Based Organization CCC Christian Community Church CCC-ABET Christian Community Church Action for Behavior Transformation CD Country Director CHSU Community Health Sciences Unit COP Country operational plan COPRED Community Partnership for Relief and Development COVISODE Common Vision for Social Development CSI Child Status Index CSW Commercial Sex Workers CTO Contracting Technical Officer DBS Dry Blood Spot DCD Deputy Country Director DHO District Health Office DQA Data Quality Assessment FBO Faith Based Organization FOCUS Foundation for Community Support Services GFTAM Global Fund Tuberculosis, Aids & Malaria GOM Government of Malawi HBC Home-Based Care HRM Human Resource and Management HTC HIV Testing and Counselling IDU Injectable Drug Users IEC Information, Education and Communication IGAs Income Generating Activities INGO International Non-Governmental Organization ISP Institutional Strengthening Plan ISPE In-School Peer Educators M&E Monitoring and Evaluation MACRO Malawi AIDS Counselling and Resource Organization MAICC Mponela AIDS Information Counselling Center MANERELA Malawi Network of Religious Leaders Living with or Affected by HIV/AIDS MARPS Most-At-Risk Populations MBCA Malawi Business Coalition against HIV and AIDS M-CAT Management Capacity Assessment Tool MCH Maternal and Child Health MDHS Malawi Demographic Health Survey MER Monitoring, Evaluation, and Reporting MIAA Malawi Interfaith AIDS Association MMM (3Ms) Mai, Mai, Mwana MoH Ministry of Health MPCBE Multi-Partner Capacity Building Event MSM Men who have Sex with Men NAC National AIDS Commission NACC Namwera AIDS Coordinating Committee NAF National Action Framework NASO Nkhotakota AIDS Support Organization NGO Non-Governmental Organization NMCM Nurses and Midwives Council of Malawi OCA Organizational Capacity Assessment OD Organization Development OSPE Out-Of-School Peer Educators OVC Orphans and Vulnerable Children PACAM Palliative Care Association of Malawi PC/HBC Palliative Care/ Home Based Care PEP Post Exposure Prophylaxis PEPFAR President‘s Emergency Plan for AIDS Relief PIH Partners in Hope Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 6 PLHIV People Living with HIV PLWA People Living With AIDS PMP Performance Monitoring Plan PMP Performance Monitoring Plan PMTCT Prevention of Mother to Child Transmission of HIV Pre-ART Pre- Antiretroviral Therapy PVC Private Voluntary Cooperation PwP Prevention with Positives REACH Rapid and Effective Action Combating HIV and AIDS RFAs Request For Applications RRM Rapid Response Mechanism SAT Southern African AIDS Trust SGFM Senior Grants and Finance Manager SWAM Society of Women and AIDS in Malawi TIs Targeted Interventions TOR Terms of Reference TWG Technical working group UNAIDS Joint United Nations Programme on HIV/AIDS UNGASS United Nations General Assembly Special Session UNICEF United Nations Children‘s Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 7 ACKNOWLEDGEMENTS The evaluation consultants would like to sincerely thank all the people that participated in the consultations and evaluation activities of the Pact Malawi Community REACH Program. The team is grateful to Mrs. Ndasowa Chitule and Chimwemwe Chitsulo of USAID Malawi for their support throughout the implementation of the evaluation assignment. Thanks go to Kate Musimwa the Country Director of Pact Malawi and all her management and staff for their input through one-on-one and team meetings and through responding to questionnaires. Rolex Tolani and Cecilia Maganga of Pact Malawi deserve special appreciation for their support and sub-partner mobilization during the field consultations for data collection from staff of partner organisations, district and community stakeholder. Thanks to all community members that participated in Focus Group Discussions during the field work. Their information was very valuable and contributed greatly to the evaluation findings and report. Gratitude goes to all Pact Malawi stakeholders and all those that contributed to the evaluation and helped in one way or another. Zikomo kwambiri Thank you very much. Salephera Consulting Ltd Private bag 152 Lilongwe Malawi Tel: +265 (0) 888828740 E-mail: hesternb@gmail.com Website: www.salephera.com Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 8 EXECUTIVE SUMMARY This document is a report of the Pact Malawi Community REACH program evaluation carried out by Salephera Consulting Ltd between October 2011 and December 2011. For five years (2007 to 2011), Pact implemented a rapid response mechanism program for HIV and AIDS in Malawi, supported by the US President‘s Emergency Plan for AIDS Relief (PEPFAR) through the US Agency for International Development in Malawi (USAID/Malawi). The program was part of Pact‘s global Community REACH (Rapid and Effective Action Combating HIV and AIDS) Leader with Associate Award, which sought to empower local organizations to support HIV and AIDS programs that reach individuals, families, and communities most vulnerable to HIV infection and HIV-related consequences with the services they need most. The Community REACH project focused on providing grant award and administration services that allowed the U.S. Mission in Malawi to increase the resources available to local organizations implementing programs that address key technical focus areas of the President‘s Emergency Plan for AIDS Relief (PEPFAR) and USAID Malawi‘s initiative to integrate HIV and AIDS activities into the Primary Health Care System. Pact Malawi, as a recipient of the award, was responsible to provide sub-granting, organizational development and technical assistance in HIV/AIDS prevention, care, and support programs. Pact Malawi‘s sub partners implemented activities focusing on prevention of HIV, HIV Testing and Counselling, Home Based Care or Palliative Care, Prevention of Mother to Child Transmission of HIV, Orphans and Vulnerable Children and Medical Injection safety. USAID Malawi mission actively participated in the implementation of the program and was involved in the selection of sub-partners for all the three Annual Program Statements (APS) from 2007, 2008 to 2009. A total of 25 sub-partners were successfully selected and supported. USAID followed the progress of the Community REACH program through site visits, reports, regular and partner review meetings. These meetings provided forums for organizational networking, sharing of achievements and experiences and identification of ways of further strengthening of sub-partner programs and organizations. The Pact Malawi Community REACH program registered impressive and significant achievements over the five year period of its implementation. Several commendable innovations and best practices arose from the Pact program including community mobilisation for PMTCT, Male Championship Groups, Quality Standards for OVC care and others. These innovations and best practices were possible because the local partners were seeking to achieve the best for their communities. The values of these innovations included adoption and replication by the government of Malawi and other similar development partner financed programs. The following were some of the achieved results of the Pact Malawi Community REACH program: A total of 72,590 pregnant women underwent HIV testing and counselling within the context of PMTCT at Antenatal Clinics (ANC) supported by sub-partners under the program. This was a very important step in identifying ART eligible clients among the pregnant women. As a result a total of 4,723 HIV positive pregnant women received ART through the Community REACH program and 91% of the babies born to these mothers tested negative. This greatly increases the survival rate of the babies born to HIV infected women. A total recorded number of 177,400 orphans and vulnerable children received care and support thereby reducing the burden on the affected families and communities. A total of 2,331,115 million individuals received prevention messages through the HIV/AIDS prevention strategy of the Community REACH program in Malawi. In light of Pact‘s achievements Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 9 of 3 million worldwide in HIV/AIDS prevention, the Pact Malawi‘s program contributed a remarkable two thirds. Over 85,000 home based care clients were supported thereby contributing to the improvement of quality of life of those receiving palliative care in Malawi. A total of 44,944 health care workers and community volunteers were trained in various HIV topics. This enhanced the capacities and capabilities of this cadre of frontline staff in their skills and readiness to respond to the challenge. The emphasis on systems strengthening, a community based approach to health care and the use of volunteers had a great impact on the type of results that Community REACH program delivered (as in achievements outlined above). These results and achievements indicate that Pact Malawi‘s Community REACH program significantly contributed to the plans and strategies of the Malawi Government in combating the HIV pandemic. The program also significantly made contributions to the achievement of PEPFAR‘s objective of reaching more local beneficiaries with HIV and AIDS services in a quick and efficient manner. With regards to sub-partner‘s progress and growth, many of the Malawian partners advanced and moved from the nascent stage (embryonic) to growing as well as expanding (growing beyond maturity) stage of organizational capacity. With reference to the features of the local organisations that improved in the Organisational Capacity Assessment (OCA) and re-OCA, the following specific growth characteristics were demonstrated: Out of 22 partners 15 improved in strategic planning; 11 out of 22 partners improved in governance, leadership and change; 12 out of 22 partners improved in Material and Financial Resources; 11 out of 22 partners improved in Organizational Culture; 13 out of 22 partners improved in Human Resources Management; 13 out of 22 partners improved in Knowledge Management;16 out of 22 partners improved in external Partnerships and Networking; 14 out of 22 partners improved in Program Management; 11 out of 22 partners improved in Administration and 12 out of 22 partners improved in Advocacy and Lobbying. This growth has potential of enhancing better service delivery of HIV and AIDS services to local communities through the local NGOs, CBOs and FBOs. Some of the local Pact Malawi partners including NASO, Malamulo, NACC, COPRED, CCC, MAICC, Ekwendeni Hospital and Tovwirane, developed capacity and capability to write proposals to solicit more funding from other sources (See conclusion 1 below). A unique feature of the Community REACH program was the inclusion of organizational capacity development with grant making and technical support to sub-partners. This enhanced the contributions of the local organizations to meeting the goals of various government ministries and departments that they work with across the Malawi nation. CONCLUSIONS 1. Pact Malawi Community REACH program helped USAID Mission to provide a platform to grow and build capacities of local organizations. As a result of the capacity growth, some of the sub￾partners developed the ability to mobilize resources directly from different financing agencies. For example, MAICC resourced additional funding from Concern World Wide and Care Malawi, Tovwirane from Plan Malawi, NACC and NASO from Firelight and also NACC from National AIDS Commission, COPRED from HIVOS, Matunkha from friends of Matunkha in Netherlands. Consequently, the sub-partners that implemented this program developed the potential of becoming bigger, better and well able to deliver high quality services. This development and evidence of growth will contribute to the sustainability of the local organizations and their work in Malawian rural and peri-urban communities. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 10 2. Pact Malawi‘s Community REACH program significantly contributed to the national response: the program was key in informing the Government of Malawi‘s plans and strategies for community-level approaches to combating HIV and AIDS; the program also significantly contributed to the achievement of PEPFAR objective of reaching more local beneficiaries with HIV and AIDS services in a quick and efficient manner. 3. The Community REACH program in Malawi gave evidence of the importance of continued capacity building of local organizations as a means of enhancing the sustainability and increased local ownership of development programs. Improvements in the organizational capacity of Pact sub-partners were due to strengthened core competencies of these local organizations to deliver strong technical programs and increased capacity to manage increasing resources. 4. The Community REACH program has had substantial success in rapidly enhancing the response to HIV/AIDS in Malawi at community level. Pact‘s activities provided a solid base in terms of human resources and infrastructure in different local organizations and in the community from which to expand USAID Malawi‘s care and support initiatives. 5. Pact‘s Community REACH program largely attained its targets for persons that received capacity development and technical assistance and also people that received the services, including scaling up sites providing PMTCT, HTC, HBC and Pre-ART services. 6. Pact‘s support to its partners demonstrated that a local NGO with sound and transparent financial management will build confidence in the community it is serving as well as attracting local and international donors to fund it. 7. Networking and collaboration among Pact Malawi partner organizations was a key for success. Networking provided avenues for sharing strategies and best practices. Service providers were able to acquire and enhance their service delivery skills, use appropriate and relevant techniques; through increased awareness of each other‘s roles and activities, local program implementers were able to minimize duplication of interventions. 8. Pact Malawi achieved excellent working relations with the MOH and other stakeholders at national, regional and local levels. While these relations enabled Pact Malawi to make significant contributions to achieving its targets, they also fostered, to a lesser degree, feelings of dependency by few sub - partners. Factors for the dependency included limited financial support sources (USAID was the main finance contributor for most of the partners); medical supplies were mostly from the ministry of health that was also facing serious shortages. The follow-on activity must address this dependency from the onset and articulate a clear exit strategy in the design. 9. Pact Malawi‘s establishment of a monitoring and evaluation department and system enabled organizations to keep better track of program data. It also allowed for data to be more easily accessed and useful. It is essential that monitoring data be used to inform program decision making. However, the absence of baseline data for HIV and AIDS outcome level indicators made it difficult to measure progress against benchmark indicators. As a result progress was measured against estimated targets by Pact Malawi staff and their partner agencies‘ counterparts. 10. The capacity building efforts of partner organisations by Pact Malawi illustrated that the development of a strategic plan creates organizational focus and draws staff to understand activities that are planned and reasons they are necessary. Strategic plans kept sub-partners on track and were used as tools to mobilize resources. 11. Pact Malawi capacity development support that combined both technical as well as organizational development proved effective in improving local organisations. OD interventions supported partners to improve their organizational systems, leadership and management, strategic planning, governance and monitoring and evaluation among others. These were areas of great need at the Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 11 beginning of the program. The technical aspects improved their service delivery skills. By combining the two, the interventions became richer and more effective. 12. Formation of data quality teams by Pact Malawi partners which constituted cross functional teams made up of staff, local leaders and other stakeholders with different levels of responsibility made the results and achievements of partners rich because this promoted shared ownership and responsibility for ensuring that the compilation and reporting of results was accurate and well documented. RECOMMENDATIONS 1. USAID should consider providing direct funding to Pact Malawi partners whose capacities reached a level of maturity (these include organizations such as NASO, Malamulo, NACC, COPRED, CCC, MAICC, Ekwendeni Hospital and Tovwirane). These agencies had teams of qualified personnel with vast experience working with many donors. They had functional boards of trustees and sound financial and administrative systems. 2. USAID and Pact Malawi should facilitate on-going support and mentorship of the mature organisations in areas such as a) building research capacity, b) leading and strengthening networks and c) negotiating with donors. This can be an on-going role for Pact Malawi. 3. For future projects Pact Malawi will need to lobby the government of Malawi for increased availability of supplies such as IEC materials, condoms and test kits. Most at risk population groups as well as people living with HIV/AIDS should be provided with steady supply of condoms to reduce the risk of contracting HIV or re-infection among the PLHIV. This level of supply quantities was beyond the capabilities and capacities of sub-partners. 4. Future USAID-funded projects should seek to include a health systems strengthening component directed at strengthening commodity availability and security. It was observed that Pact Malawi Community REACH Program successfully created demand for services but the efforts were undermined by intermittent drug supplies. 5. In future, PEP should be modelled around the community PMTCT concept which proved to be effective and with functional follow–up procedures and where defaulters are quickly identified and brought back to the treatment. 6. For the next generation of indicators of care, there is need to split the beneficiaries of OVC and PC/HBC. This will necessitate understanding of how many people have benefitted from OVC or PC/HBC. 7. For sustaining program activity, Pact Malawi partners should continue to provide HTC outreach where at-risk youth congregate, or where other most at-risk persons live or work, to increase uptake by those most in need of care, support and prevention interventions. In addition, peer to peer HIV prevention education should be scaled up. 8. Future program design USAID should include baseline assessments to establish HIV and AIDS outcome indicators benchmarks in order to track progress against set objectives, outcome and impact indicators. 9. Capacity development initiatives require resources in terms of time, material, human and financial. All capacity development initiatives in the future should be attached to budgets to avoid the situation where by partners plan for the initiatives but never implement them because of lack of resources. This limitation should be taken into account in the design of future programs of similar nature to the Pact Malawi Community REACH. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 12 SECTION ONE 1.0 INTRODUCTION AND BACKGROUND This document is a report of the Pact Community REACH program evaluation carried out by Salephera Consulting Ltd between October 2011 and December 2011. For five years (2007 to 2011), Pact implemented a rapid response mechanism program for HIV and AIDS in Malawi, supported by the US President‘s Emergency Plan for AIDS Relief (PEPFAR) through the US Agency for International Development in Malawi (USAID/Malawi). The program was part of Pact‘s global Community REACH (Rapid and Effective Action Combating HIV and AIDS) Leader with Associate Award, which sought to empower local organizations to support HIV and AIDS programs that reach individuals, families, and communities most vulnerable to HIV infection and HIV-related consequences with the services they need most. The Community REACH project focused on providing grant award and administration services that allowed the U.S. Mission in Malawi to increase the resources available to local organizations implementing programs that address key technical focus areas of the President‘s Emergency Plan for AIDS Relief (PEPFAR) and USAID Malawi‘s initiative to integrate HIV and AIDS activities into the Primary Health Care System. Pact Malawi, as a recipient of the award, was responsible to provide sub-granting, organizational development and technical assistance in HIV/AIDS prevention, care, and support programs. Pact Malawi sub partners implemented activities focusing on prevention of HIV, HIV Testing and Counselling, Home Based care or Palliative Care, Prevention of Mother to Child Transmission of HIV, Orphans and Vulnerable Children and Medical Injection safety. Pact Malawi, therefore, provided a multi-layer program of grants management, targeted, appropriate HIV/AIDS technical assistance and organizational capacity building for local organizations in Malawi; The Community REACH project ran from January 2007 through September, 2011 (with a further 3 months extension to December 2011 for a close out phase). The evaluation report will be used by Pact to assess the extent to which its capacity building model succeeded in providing technical assistance and enabling local NGOs, CBOs, and FBOs to grow as organizations, manage larger programs and access other additional funding. The specific recommendations and lessons learned documented in this evaluation report will also be used by the USAID Malawi Mission to determine how future capacity building programs can be more effectively designed. 1.1 HIV/AIDS SITUATION IN MALAWI Malawi is one of the ten poorest nations in the world with over half of the population living on $1 per day or less. HIV/AIDS prevalence rates are high making Malawi one of the ten countries most affected by AIDS worldwide. Out of a population of 15.4 million, almost one million people in Malawi are living with HIV. 1 2AIDS is the leading cause of death amongst adults in Malawi, and is a major factor in the country‘s low life expectancy of just 54.2 years. 3 1 UNAIDS (2010) 'UNAIDS Report on the Global AIDS Epidemic' 2 UNDP (2011) ‗Human Development Report 2011‘ 3 UNDP (2011) ‗Human Development Report 2011 Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 13 The AIDS crisis has affected all sectors of society in Malawi, but certain patterns have emerged as the epidemic has progressed: The majority of HIV infections in Malawi occur through heterosexual sex. 4 There is no available information about the number of infections transmitted through sex between men, as homosexuality is illegal, but indications from small-scale studies suggests prevalence may be as high as 21.4% among MSM. 5 6 There is a higher rate of HIV prevalence amongst women than amongst men: around 60% of adults living with HIV in Malawi are female. 7 The majority of HIV infections occur amongst young people, particularly those between the ages of 13 and 24. 8 The epidemic has heavily affected children. In 2009 an estimated 120,000 children in Malawi were living with HIV, and more than half a million children had been orphaned by AIDS. 9 HIV prevalence is around 17 percent in urban areas, compared to almost 11 percent in rural areas.10 However, studies suggest that prevalence is declining in many urban areas and rising in many rural ones. 11 There is a high HIV prevalence amongst certain labour groups in Malawi, including sex workers (70.7 percent), female police officers (32.1 percent) and male primary school teachers (24.2 percent). 12 The single biggest constraint on health services is the shortage of health professionals throughout the country, emphasizing the importance of the civil sector to address HIV/AIDS responses. For example there is 1 doctor for every 50,000 people in Malawi. The national response to HIV/AIDS is coordinated by the National AIDS Commission (NAC), located in the Office of the President and Cabinet. Malawi developed the National HIV/AIDS Strategic Framework in 1999 and it was launched by the president in 2004. The Government of Malawi (GOM) implemented its Second National HIV/AIDS Action Framework for 2005-09 building from the groundwork laid by the first one. The key priorities included: 1) prevention and behavior change; 2) treatment, care and support; 3) mitigation: socio-economic and psychosocial; 4) mainstreaming, partnerships and capacity building; 5) research and development; 6) monitoring and evaluation; 7) resource mobilization, tracking and utilization; and 9) national policy coordination and program planning. The process of developing the national HIV/AIDS policy was very consultative, with representatives from the public sector, faith-based organizations, employers‘ associations, trade unions, the media, health workers and PLHIV participating in the consultations. The objectives of the policy are to guide the national response to HIV/AIDS, both in order to prevent further HIV transmission and to mitigate the impact of HIV/AIDS. The major areas of intervention include continuing with relentless implementation of the comprehensive multi-sectoral response to HIV/AIDS; HIV prevention, treatment, care and support; the protection, participation and empowerment of PLHIV and vulnerable groups; traditional and 4 WHO/UNAIDS/UNICEF (2011) ‗Global HIV/AIDS Response: Epidemic update and health sector progress towards Universal Access 2011 5 Beyrer, C et. al (2010, August) 'Bisexual concurrency, bisexual partnerships, and HIV among Southern African men who have sex with men' Sex Transm Infect 86(4) 6 "A cross-sectional evaluation of the HIV prevalence and HIV-related risk factors of men who have sex with men (MSM) in Malawi", E. Umar, et al. AIDS 2008 - XVII International AIDS Conference: Abstract no. MOPE0412 7 UNAIDS (2008) 'Report on the global AIDS epidemic' 8 UNAIDS (2008) 'Report on the global AIDS epidemic' 9 'UNAIDS (2010) 'UNAIDS report on the global AIDS epidemic' 10 UNGASS Country Progress Report (2010) 'Malawi HIV and AIDS Monitoring and Evaluation Report: 2008-2009' 11 G A Bello, J Chipeta and J Aberle-Grasse (2006), 'Assessment of trends in biological and behavioural surveillance data: is there any evidence of declining HIV prevalence or incidence in Malawi?' 12 UNGASS Country Progress Report (2010) 'Malawi HIV and AIDS Monitoring and Evaluation Report: 2008-2009' Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 14 religious practices and services; HIV/AIDS in the workplace; establishing a national HIV/AIDS research agenda; and monitoring and evaluation. In spite of this impressive early collaboration among Government, international bilateral partners, NGO/FBOs, and the private sector, there are serious structural and capacity barriers for continued scale-up. Providing comprehensive HIV/AIDS services to a mostly rural population requires a robust decentralized, community-based approach supported by strong policies and leadership from the central level. Poverty poses a major challenge with household food security and limited access to services for the majority of the population. Recent global financial crises have led to reduced availability of financial resources for HIV and AIDS with development partners transferring responsibility to national governments for mobilising their own resources. In recent years, Malawi has faced challenges in securing access to Global Fund financial resources. A reduction or scarcity of financing for the national response is likely to negatively impact the progress the country has made over the years. 1.2 UNITED STATES OF AMERICA GOVERNMENT (USG) HIV/AIDS STRATEGY In the 2006 -2010 PEPFAR Malawi Strategic plan, the U.S. Mission in Malawi had the vision to expand and enhance its HIV/AIDS programs so that by 2010 Malawians would have more comprehensive access to high quality HIV/AIDS prevention, care and treatment services. The principal actions for implementing that strategic plan included: 1. Supporting prevention programs for populations at risk, including youth; 2. Increasing and encouraging voluntary testing and counselling (VCT); 3. Supporting efforts to ensure all those eligible for antiretroviral therapy (ART), including Prevention of Mother to Child Transmission (PMTCT), are provided with appropriate drugs and related support and care; 4. Developing and supporting national systems to support People Living with AIDS (PLWA); 5. Supporting palliative care for individuals TB, HIV/AIDS co-infected; 6. Supporting and developing national palliative care standards; 7. Strengthening local community capacity, including faith-based community structures; 8. Expand human capacity through technical assistance, training, supportive supervision, and mentoring of key health care providers; 9. Supporting development of strategic information systems at organizational, district and national levels; 10. Improving distribution systems of drugs, reagents and other supplies to health facilities throughout the nation; and 11. Encouraging coordination with other collaborating partners while maintaining focus on the objectives and principles of the Emergency Plan. In 2009, the U.S. Mission in Malawi developed a follow on strategy document to the 2006 strategic plan, the Partnership Framework for implementing the PEPFAR program in Malawi from the period 2009 - 2013. The partnership framework addresses the HIV/AIDS pandemic in the context of the new developments that have taken place between 2006 and 2010 in HIV/AIDS programming. Among others, key developments include; a reduction of the HIV prevalence rate from 12% in 2004 to 10.6% (MDHS), emerging evidence demonstrating new key drivers of the HIV/AIDS epidemic in Malawi such as Multiple Concurrent Partnerships, most at risk populations and the tremendous scaling up of ART, HTC, PMTCT and other key HIV/AIDS services to the Malawian population. The period has also seen a transition in the levels of HIV/AIDS knowledge in the Malawian population and hence HIV/AIDS awareness programs have evolved to focus more on behavior change interventions, more comprehensive HIV/AIDS knowledge and skills beyond just awareness. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 15 Another crucial development and change within the 2006 - 2010 period was the changing global financing environment which has seen less and less available resources for HIV programs in the face of ever increasing HIV treatment and care costs as programs successfully scaled up and this development has also called for more host government ownership to finance the epidemic in their countries. Within this changing environment, the U.S. Mission‘s 2009 -2013 Partnership Framework identified the following key objectives to address the HIV/AIDS pandemic in Malawi: a. Reduce New HIV infections in Malawi Extensive knowledge now exists in Malawi both about the dynamics of the HIV epidemic as well as the nature and structure of sexual networks, multiple concurrent partnerships, and other high risk behaviors. Nevertheless, translating high knowledge into accurate individual risk perception and effective strategies for reduction of concurrent partnerships is expected to be a key area of focus in future programming. Increased coverage of targeted prevention activities for most-at-risk populations is also required, with attention to other vulnerable groups, who have not been reached in past interventions. b. To provide and expand equitable treatment for PLHIV and mitigate the health impact of HIV and AIDS Significant progress has been made in the national ART, PMTCT and HIV Testing and Counselling (HTC) programs. Improvements in supply chain are also important to ensure an uninterrupted treatment and care programs. c. To mitigate the economic and psychosocial effects of HIV and AIDS and improve the quality of life for PLHIV, OVC, and other affected individuals and households The national response has targeted OVCs, the Elderly and other vulnerable groups with impact mitigation type of interventions. These activities need to be addressed in a more comprehensive manner. d. To increase the involvement and contribution of public sectors, private sector and civil society in the HIV and AIDS response Civil society organizations and local assemblies play a critical role in implementing the national programs in prevention, treatment, care and support and impact mitigation. Continued financial and capacity building support to these groups remains a high priority. GOM Ministries have implemented workplace programs but there is need to address the program‘s weaknesses in HIV mainstreaming by prioritizing line ministry support. e. To generate and disseminate information about the HIV and AIDS epidemic and response, to inform appropriate policy and evidence-based practice. Malawi continues to be on the cutting edge for HIV and AIDS programming and has included an annual research dissemination meeting to share best practices and challenges in program implementation. f. To support enhanced HIV and AIDS financial resource mobilization and management at all levels g. To support facilitation and coordination of the multi-sectoral implementation of the national HIV and AIDS action framework The Partnership Framework has an underlying implementation strategy to support achievement of the above mentioned objectives. The U.S. Mission in Malawi implements these objectives and ensures Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 16 sustainability through the following strategies: i) working in full alignment with the GOM priorities and ii) strengthening health systems including civil society and local communities. USAID/Malawi, as a key implementing agency under the U.S. Mission in Malawi, has therefore set priorities consistent with its own principles, which include using USAID‘s strengths in HIV/AIDS programing, applying lessons learned from recent experiences in Malawi, and ensuring greatest impact consistent with funding streams and levels. Per the agreed Partnership Framework document, USAID/Malawi implements HIV/AIDS programs with an emphasis on prevention including behavior change communication, care and support, OVC, and PMTCT. USAID‘s health interventions also place a strong focus on capacity building, institutional strengthening, and technical assistance for local organizations as a sustainability measure as well as a mode of enabling resources to reach more Malawians and marginalized populations. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 17 SECTION TWO 2.0 PURPOSE, METHODOLOGY AND STRATEGY OF THE EVALUATION The purpose of this evaluation was threefold; 1. To review, analyze and evaluate the extent to which the Community REACH project achieved its stated objectives. 2. To assess Pact‘s capacity building model‘s success in providing technical assistance and enabling local NGOs, CBOs, and FBOs to grow as organizations, manage larger programs and access other additional funding. 3. To provide specific recommendations and lessons learned that the Mission can explore and utilize in designing future capacity building programs. The evaluation mission reviewed, assessed and provided documented qualitative and quantitative findings and recommendations of the Community REACH project in Malawi covering the period 2007 to 2011. Qualitative and quantitative aspects of the program design and results were considered and areas in the program were identified that have had the most and the least improvements giving reasons why and resulting effects on capacity building and HIV programming in Malawi. 2.1 EVALUATION METHODS AND STRATEGIES The following tasks were completed in the implementation of the evaluation: The evaluation mission conducted in-brief and debrief meetings with USAID/Malawi. A four member team conducted the evaluation and took site visits to the agreed locations to interview key informants from the main national and international stakeholder agencies, and other NGOs working in the same areas. Interviews were designed to answer the questions posed in the TORs (See annex 8) and identify factors associated with success and the achievement of program objectives. The evaluation team used both qualitative and quantitative methodological approaches to gather information and analyze Community REACH Program performance. Extensive literature review was done of key program documents, appropriate national documents and semi-annual reports of Pact Malawi. Evaluations of OCAs and re-OCAs were also used. For QUANTITATVE data collection from stakeholders, a special tool (questionnaire) for semi￾structured interviews was developed with specific questions and unique areas of assessment included for isolating information that is agency specific and not generic. Pact Malawi‘s semi-annual and annual reports were used for aggregating data that was analysed and presented in the various sections of this report. For QUALITATIVE data collection, a checklist of questions was developed for guiding Focus Group Discussions with Pact staff, community members, grantees and other entities including participating structures at community level. Special protocols for Focus Group Discussions and Key Informant Interviews were developed and administered by our professional team of consultants. (See annexes 3 to 7). Tools Translation and Back Translation: Where required and appropriate, evaluation tools were translated from English to local language. For quality control, the translated tools were back translated by an independent communication‘s expert to ensure that meaning and intent of the tools were not lost. This is best practice and ensures high quality data collection, and subsequently good analysis and report. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 18 In-depth/Key informant interviews The evaluation mission held in-depth interviews with the staff at the Pact Malawi Community REACH Program office in Lilongwe. The team also held interviews with NGO managers, trained educators, local promoters, community leaders, and beneficiaries at project locations. Semi-structured interviews Semi structured interviews were held with selected grantees and representatives of stakeholder institutions from government, civil society, and the international community, including MoH, Global Fund, Bridge Project and the NAC. Sampling and Field visits Field visits were carried out to NGO grantees that work in project areas of Community REACH Program. Community REACH Program assisted with selection of Sub-grantees and sites and helped the scheduling of field visits. The following criteria were used in the sampling: Funding level received by the sub partner (ranging from high to low) Length of support received from Pact (ranging from shortest to longest) Type of CBO (representation across FBO, local CBOs, national networks and others) Technical areas covered (At least all technical areas will be covered in the sample) Geographical location by Malawi‘s regions (South, Centre and North) Table 1 : Pact Sampled Community REACH program sub-partners across the three regions of Malawi: REGION SUBGRANTEE Location Type NORTH 1. Lusubilo Karonga FBO 2. FVM Matunkha Rumphi FBO 3. Tovwirane Mzimba NGO 4. Ekwendeni Mzimba / Mzuzu FBO CENTRE 1. Society of Women and AIDS in Malawi (SWAM)13 Nkhotakota NGO 2. Nkhotakota AIDS Support Organization (NASO) Nkhotakota NGO 3. Mponela AIDS Information and Counselling Centre (MAICC) Dowa NGO 4. Nkhoma Mission Hospital (CCAP) Lilongwe FBO 5. Partners in Hope Lilongwe NGO 6. Light House Lilongwe NGO 7. Nurses and Midwives Council of Malawi Lilongwe Parastatal 8. National Association of People Living with HIV and AIDS in Malawi (NAPHAM) Lilongwe NGO SOUTH 1. Christian Community Church (CCC) Thyolo NGO 2. Community Partnership for Relief and Development (COPRED) Blantyre NGO 3. Malamulo Thyolo FBO 4. Namwera AIDS Coordination Committee (NACC) Mangochi NGO 5. Adventist Health Services (AHS) Blantyre FBO 13 This was the organisation selected for the pre-testing of the evaluation tools. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 19 Evaluation Limitations 1. Lack of baseline survey for outcome level indicators for HIV and AIDS at the beginning of the Pact Malawi program, made it difficult to measure progress against the benchmarks. The project measured progress against estimated targets. 2. During the period of the evaluation, it was found that some partner agencies had laid off key staff for the program and this presented a challenge in data collection process. This was mitigated by the partner agencies making special arrangement for the particular staff to come for the interviews. This led to the evaluation mission delaying due to waiting. 3. Field travel was seriously impacted by lack of fuel and long delays as a result of long queues when need arose for refuelling. This led to a prolonged evaluation period as many appointments had to be cancelled and rescheduled. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 20 SECTION THREE 3.0 FINDINGS ON PACT COMMUNITY REACH PROGRAM DESIGN In Malawi Pact achieved Community REACH purpose by offering competitive grant awards and comprehensive technical and organizational capacity building assistance to national NGOs, CBOs, and FBOs. This support enhanced best practices and innovative approaches in the local organisations that partnered with Pact Malawi implementing community-based HIV and AIDS prevention, care, and support activities. This $20 million USAID Malawi grant to Pact Malawi complemented the Government of Malawi‘s and the US Government‘s efforts in responding to the following seven priority areas: a) Prevention and Behavior Change b) Treatment, Care and Support c) Impact Mitigation d) Mainstreaming and Decentralization e) Research, Monitoring and Evaluation f) Resource Mobilization and Utilization g) Policy and Partnerships Below is a complete list of the local organisations that Pact Malawi supported between 2007 and 2011:14 1. Adventist Health Services (AHS) (Closed out in 2010) 2. Christian Community Church (CCC) (Closed out in 2010) 3. Common Vision for Social Development (COVISODE) (Closed out in 2010) 4. Community Partnership for Relief and Development (COPRED) 5. Ekwendeni Mission Hospital 6. Foundation for Community Support Services (FOCUS) (Closed out in 2010) 7. FVM-Matunkha 8. Lighthouse Trust (Closed out in 2010) 9. Lusubilo 10. Malamulo Hospital 11. Malawi AIDS Counselling Resource Organization (MACRO) (Closed out in 2010) 12. Malawi Business Coalition Against HIV and AIDS (MBCA) (Closed out in 2010) 13. Malawi Interfaith AIDS Association (MIAA) (Closed out in 2010) 14. Mponela AIDS Information and Counselling Centre (MAICC) 15. Namwera AIDS Coordination Committee (NACC) 16. National Association of People Living with HIV and AIDS in Malawi (NAPHAM) 17. Nkhoma Mission Hospital 18. Nkhotakota AIDS Support Organization (NASO) 19. Nurses and Midwives Council of Malawi 20. Palliative Care Association of Malawi (PACAM) (Closed out in 2010) 21. Partners In Hope (PIH) 22. Society for Women Against HIV and AIDS in Malawi (SWAM) 23. Southern African AIDS Trust (SAT) (Closed out in 2010) 24. Tovwirane 25. Tutulane (suspended in December 2008 due to unsatisfactory fiscal discipline) 26. Zomba Catholic Health Commission (suspended in April 2010, following identification of deficiencies in program quality.) 14 See Map on the next page for the location of each PACT partner in Malawi Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 21 Figure 1: Map of Pact partners Geographic Location and services delivered Karonga MACRO: HTC Lusibilo: OVC, HBC FOCUS: OP AHS: Prev. HTC, PMTCT Chitipa AHS: Prev. HTC Rumphi NAPHAM: Prev., HBC, OVC, AB, OP, HTC FVM –Matunkha: AB, OP, OVC, PC/HBC AHS: Prev. HTC, Mzimba Tovwirane: AB, HTC, OVC, HBC, PMTCT Ekwendeni: Prev., PMTCT, HBC, AB, HTC MACRO: HBC AHS: Prev. HTC, PMTCT, Paediatric Care Nkhotakota SAT: HTC, PMTCT, HBC SWAM: Prev., HTC NASO: AB, OVC, PC/HBC, Policy Dowa MAICC: HTC, HBC, OVC Mangochi MBCA: OP.HTC NACC: OVC, OP, HTC, PMTCT Machinga NAPHAM: Prev., HBC, OVC Zomba MACRO: HTC NACC: OP, HTC Blantyre COPRED: Prev., HBC, OVC MACRO: HTC AHS: Prev., HTC, PMTCT, PedC Nsanje NAPHAM: Prev.HBC, OVC SAT: Prev., OVC, PMTCT, HTC Chikwawa Malamulo: Prev., PMTCT, HTC, HBC COVISODE: AB, HTC, Policy Thyolo Malamulo: Prev., PMTCT, HTC, HBC, PedC MBCA: OP, HTC Christian Community Church: AB, OP Nkhata Bay AHS: PMTCT, HTC, AB, PedC Ntcheu AHS: PMTCT, HTC, AB, PedC Mwanza AHS: Prev., HTC, PMTCT Mulanje AHS: Prev, HTC, PMTCT, PedC Neno AHS: PMTCT KEY AB - Abstinence/Be Faithful Prevention Campaign HBC - Home Based Care HTC - Home Testing and Counselling OP - Other Prevention Messages OVC - Orphans and Vulnerable Children PC - Palliative Care PedC – Paediatric Care PEP - Post Exposure Prophylaxis PMTCT - Prevention of Mother to Child Transmission Policy – Health/HIV Policy Prev. - HIV Prevention National Malawi Interfaith AIDS Association (MIAA): AB, HTC Palliative Care Association of Malawi: PC NMCM: PEP, Injection Safety Lilongwe PIH: PMTCT, HTC, AB LIGHTHOUSE TRUST: PMTCT, HTC, AB Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 22 3.1 PACT MALAWI COMMUNITY REACH PROGRAM CONTRIBUTIONS TO THE MISSION, VISION AND STRATEGIES OF USG AND MALAWI GOVERNMENT Pact Malawi‘s Community REACH project was designed to contribute to the achievement of USAID Malawi‘s strategy and the Government of Malawi‘s National HIV and AIDS National Framework for HIV and AIDS (2005 – 2009). The Community REACH project objectives focused on providing local organizations with a seamless and efficient grant process mechanism that integrated extensive institutional strengthening and quick access to ongoing technical assistance at all stages of the grant management cycle. The institutional strengthening and technical assistance interventions aimed to improve and ensure effective and quality service delivery and to enable indigenous organizations or sub-grantees to access U.S. Government resources. This was especially important since most of the local organizations under the Community REACH project were receiving U.S. Government funding for the first time. Many of these organizations were not eligible to receive USAID funding due to the fact that they required considerable assistance in developing their organizational management systems. The Community REACH project activities were centered on the following three main objectives: 1. To provide an effective and transparent grant award and administration system for the provision of responsive, fast track grant-making assistance to organizations responding to PEPFAR. 2. To provide implementers with access to financial resources and high quality technical expertise to assist in achieving and effectively reporting results while complying with USG financial and administration requirements. 3. To expand the civil society‘s response by providing capacity building to local, regional, national, and international organizations resulting in increased capacity of organizations and networks to provide and sustain HIV and AIDS and related health services. Through the Community REACH project, Pact Malawi therefore: 1. Provided grants to facilitate rapid implementation of HIV programming. 2. Provided technical assistance to improve service delivery. Specifically, Pact worked with partners to ensure adherence to USG OGAC guidance in all programming areas. 3. Provided focused organizational capacity building activities and monitors HIV and AIDS interventions. 4. Facilitated local linkages and dynamic networks for organizational and technical learning. 5. In collaboration with national agencies, Pact Malawi built national linkages to promote technical excellence. 3.2 PACT’S APPROACH AND ACHIEVEMENTS IN MALAWI Pact Malawi's approach to combating HIV/AIDS was based on community development principles combined with best practice programming. Working with local and national partners in the civil society, public and private sectors in Malawi, Pact Malawi contributed to the building of strong community-based organizations capable of supporting sustainable and effective responses to HIV/AIDS. With technical, organizational and institutional capacity building from Pact Malawi, communities and organizations developed appropriate responses in HIV prevention, care and mitigation. Pact Malawi‘s Community REACH Program with USAID financial support was designed to facilitate the efficient flow of grant funds to CBOs, FBOs and NGOs playing valuable roles in the struggle against HIV/AIDS. The program promoted both scale-up of successful programs and start-up of new Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 23 programs with potential for demonstrable impact. This dynamic USAID funding mechanism made funds available to local organizations for HIV/AIDS grants in intervention areas encompassing the entire HIV/AIDS prevention-to-care-to-treatment Continuum in Malawi. Pact Malawi Community REACH program issued technical support grants totalling $10 million to 23 local CBOs, FBOs and NGOs. This amount was consistent with what Pact Malawi had planned to disburse in grants to its partner organisations in the country. The Pact Malawi Community REACH program registered impressive and significant achievements over the five year period of its implementation. Several commendable innovations and best practices arose from the Pact program including community mobilisation for PMTCT, Male Championship Groups, Quality Standards for OVC care and others. These innovations and best practices were possible because the local partners were seeking to achieve the best for their communities. The values of these innovations included adoption and replication by the government of Malawi and other similar development partner-financed programs. The following were some of the achieved results of the Pact Malawi Community REACH program: 1. A total of 72,590 pregnant women underwent HIV testing and counselling within the context of PMTCT at Antenatal Clinics (ANC) supported by sub-partners under the program. This was a very important step in identifying ART eligible clients among the pregnant women. As a result a total of 4,723 HIV positive pregnant women received ART through the Community REACH program and 91% of the babies born to these mothers tested negative. This greatly increases the survival rate of the babies born to HIV infected women. 2. A total recorded number of 177,400 orphans and vulnerable children received care and support thereby reducing the burden on the affected families and communities. 3. Some 2,331,115 individuals received prevention messages through the HIV/AIDS prevention strategy of the Community REACH program in Malawi. In light of Pact‘s achievements of 3 million worldwide in HIV/AIDS prevention, the Pact Malawi‘s program contributed a remarkable two thirds. 4. A total of 85,000 home based care clients were supported thereby contributing to the improvement of quality of life of those receiving palliative care in Malawi. 5. Health care workers and community volunteers numbering 44,944 were trained in various HIV topics. This enhanced the capacities and capabilities of this cadre of frontline staff in their skills and readiness to respond to the challenge. The emphasis on systems strengthening, a community based approach to health care and the use of volunteers had a greatly contributed to the success of the Community REACH program delivered. These results and achievements indicate that Pact Malawi‘s Community REACH program significantly contributed to the plans and strategies of the Malawi Government in combating the HIV pandemic. The program also significantly made contributions to the achievement of PEPFAR‘s objective of reaching more local beneficiaries with HIV and AIDS services in a quick and efficient manner. With regards to sub-partner‘s progress and growth, many of the Malawian partners advanced and moved from the nascent stage (embryonic) to growing as well as expanding (growing beyond maturity) stage of organizational capacity. This remarkable growth has potential of enhancing better service delivery of HIV and AIDS services to local communities through the local NGOs, CBOs and FBOs. Some of the local Pact Malawi partners including NASO, Malamulo, NACC, COPRED, CCC, MAICC, Ekwendeni Hospital and Tovwirane, developed capacity and capability to write proposals to Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 24 solicit more funding from other sources. Some of them reached a stage whereby they can get direct funding from USAID either as individual entities or in partnership with other organisations. A unique feature of the Community REACH program was the inclusion of organizational capacity development with grant making and technical support to sub-partners. This enhanced the contributions of the local organizations to meeting the goals of various government ministries and departments that they work with across the Malawi nation. The program helped USAID Mission to provide a platform to grow and build capacities of local organizations. As a result of the capacity growth, some of the sub-partners developed the ability to mobilize resources directly from USAID and other bilateral development partners. Thus, the sub￾partners that implemented this program have the potential of becoming bigger, better and well able to deliver high quality services. This development and evidence of growth will contribute to the sustainability of the local organizations and their work in Malawian rural and peri-urban communities. 3.3 PACT MALAWI COMMUNITY REACH PROGRAM BEST APPROACHES The following were the best practices achieved by the Pact Community REACH program in Malawi: a) Data Quality Assessments and Data Risk Assessment levels for Pact partners. b) Establishment of data quality teams at partner level whereby some government officials at district level were incorporated and this has made government staff appreciate importance of DQA for replication. This was evident in Rumphi, Mangochi and Nkhotakota districts where district officials began to implement routine DQAs model. c) Partners‘ ability to utilize knowledge and skills gained through Pact‘s M&E mentorship to improve M&E systems for projects funded by other donors. Coordinated feedback helped improve quality of partner reporting. d) Grant making process by Pact through release of APS that has now been adopted by National AIDS Commission. e) Development of minimum packages for OVC, HBC and prevention and associated checklists to monitor progress improved programming. f) Development of Community PMTCT package which was presented to Ministry of Health and Pact was given a go ahead to implement. This is the first ever comprehensive community PMTC initiative in Malawi. g) Piloting of National OVC standards for OVC whereby Pact initiated baseline community assessments which have now been adopted by National OVC Standards task force for all stakeholders to be conducting assessments before completing roll out of standards. h) Use of Child Status Index to monitor OVC outcomes. i) Active and meaningful involvement of OVC on issues that affect them in their daily lives through Child Parliament where issues were identified and discussed and resolved in a parliamentary manner. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 25 SECTION FOUR 4.0 FINDINGS ON PACT COMMUNITY REACH PROGRAM CAPACITY BUILDING DEVELOPING SUSTAINABLE LOCAL NGOS This part of the report discusses the efficiency and effectiveness of the capacity development support that Pact Malawi provided in building capacity and developing sustainable organizations that could mobilize resources independently and manage resources better and implement strong technical programs. 4.1 The extent to which Pact Community REACH program achieved its objectives related to building capacity of local NGOs in Malawi One of the objectives of the Pact Malawi Community REACH program was to expand and strengthen the Malawi Civil Society response to HIV and AIDS by providing capacity development to local organizations to increase their capacity to provide and sustain HIV and AIDS related health services. Pact Malawi placed great emphasis on the capacity building process for its grantees throughout the grant management cycle in order to increase the sustainability and effectiveness of its partner organizations. To achieve this objective, Pact Malawi implemented capacity development initiatives in collaboration with its capacity development partner Capacity Building NGO CABUNGO (a local organization that was identified by Pact Malawi through a competitive process to do most of the OD work). This was accomplished through specialized and targeted institutional strengthening interventions. Analysis of data and evidence on capacity building achievements and impact is presented in section 4.2 below. 4.2 Pact Malawi’s approach to capacity strengthening of local NGOs CABUNGO signed a Fixed Obligation Grant (FOG) of about $160,000.00 (One hundred Sixty Thousand dollars) with Pact Malawi in September, 2008. The FOG was meant to implement a program ‗Partnership for capacity building in Malawi‘. This grant to CABUNGO was awarded with an extension by Pact Malawi that CABUNGO would sub-grant MANASO the capacity building component of advocacy and lobbying. CABUNGO was tasked with five major responsibilities. These were to conduct self guided assessments to partner organizations, conduct guided action planning sessions, develop institutional strengthening plans (ISP), develop some interventions to the partners based on the ISPs and finally submit indicators. Prior to signing the grant in September, 2008, CABUNGO and Pact Malawi undertook several activities as preparatory steps for the grant. In the first week of July, 2008, CABUNGO and MANASO were trained in the use of Organization Capacity Assessment tool (OCA). Two Pact headquarters staff spent about 8 days with three practitioners from CABUNGO and two from MANASO training and mentoring the Malawian counterparts in administering OCA. It was in late September, 2008 when CABUNGO and MANASO rolled down the OCA to Pact Malawi local partners. In order to determine the capacity needs of different sub-grantees, CABUNGO conducted participatory Organizational Capacity Assessment (OCA) which was highly participatory organizational self-assessment approach designed to promote organizational learning and capacity building. The OCA tool measured ten areas of capacity namely: 1. Strategic direction, 2. Governance, Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 26 3. Leadership and change, 4. Material and financial resources, 5. Organizational culture, 6. Human resource management, 7. Knowledge management, 8. External partnerships and networking, 9. Program management and service delivery, 10. Administration and advocacy and lobbying. Pact Malawi partner organizations underwent OCA for baselines which were used to determine capacity needs of sub-grantees and the overall capacity building agenda for every organization. The results of the OCA were used to facilitate a discussion and ultimately develop a customized plan, the Institutional Strengthening Plan (ISP), which addressed the institutional needs of each sub-grantee. These plans were developed in a participatory manner with each sub-grantee. Pact Malawi, in cooperation with CABUNGO, implemented OCAs and developed ISPs for all of its sub-grantees. Out of 26 organizations, 22 organizations went through the re-OCA which provided the extent to which organizations had changed in terms of capacity in the ten OCA tool areas. On average, sub￾partners reported overall improvements in 8 capacity areas, namely Strategic Direction; Leadership, Governance and Change; Organizational Culture; Human Resource Management; Knowledge Management; External Partnerships and Networking; Program Management and Service Delivery; and Advocacy and Lobbying. This was a positive result as it indicated overall improvements in the majority of capacity areas. Only in the case of Organizational Culture (OC) the improvement was relatively low. This was most likely because in many organizations Re-OCA participants indicated that there was a good sense of teamwork. However, a pattern that came up in most organizations and negatively affected OC is the existence of a prevailing ―culture of silence‖ whereby staff members preferred to ―keep quiet‖ instead of bringing challenges and issues to the forefront. An environment of closed communications contributed to perpetuate and worsen problems in an organization. In the case of the Material and Financial Resources (MF) capacity area the major challenge that seemed to be arising for partners was that during the time of the evaluation they were facing a tight and very competitive funding environment. Due to political challenges in Malawi, some funders are pulling out of the country or reducing development assistance. This is affecting direct funding and technical support to civil society as well as the support civil society receives from government since the government funds coming from direct assistance from donors are draining and government, in turn, has fewer resources to support civil society. However an assessment of the partner organizations has revealed that NGOs with robust financial systems in place have potential of attracting local and international donors regardless of the tight and competitive funding environment. For example NASO has managed to increase its donor base even after Pact‘s funding and it‘s the only NGO in Nkhotakota that is sub-granting to Community based organizations. With reference to figure 2 below, and as an illustration of the features of the local organisations that improved in the OCA and re-OCA, the following specific growth characteristics were demonstrated: Out of 22 partners 15 improved in strategic planning; 11 out of 22 partners improved in governance, leadership and change; 12 out of 22 partners improved in Material and Financial Resources; 11 out of 22 partners improved in Organizational Culture; 13 out of 22 partners improved in Human Resources Management; 13 out of 22 partners improved in Knowledge Management;16 out of 22 partners improved in external Partnerships and Networking; 14 out of 22 partners improved in Program Management; 11 out of 22 partners improved in Administration and 12 out of 22 partners improved in Advocacy and Lobbying. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 27 Figure 2: Sub-partners indicating improvements per capacity area NB: SD (Strategic Direction), GL (Governance and Leadership, MF (Material and Financial Resources), OC (Organizational Culture), HRM (Human Resources Management), KM (Knowledge Management), PN (Partnership and Networking), PM (Program Management), A (Administration), AL (Advocacy and Lobbying). As depicted in figure 3 below, average OCA and Re-OCA scores indicated that 13 out of 22 organizations experienced overall improvements while 9 out of 22 organizations experienced overall setbacks. An analysis of the improvements seen by each organization indicated that 21 out of 22 organizations realized improvements in at least one area. The areas with the largest number of organizations that improved were Strategic Direction, External Partnerships and Networking and Program Management. The areas with the least number of organizations improving were Governance, Leadership and Change and Administration. On average, there were positive changes in the majority of partner organizations with changes being primarily a function of internal leadership. Those organizations that showed the highest concern with quality and organizational strengthening showed the most improvement. It is important to point out that while OCA was powerful in terms of promoting organizational dialogue and helping to identify general trends, the tool had some challenges. The OCA tool was a participatory self-assessment facilitated by an external person. The facilitator did not influence the discussion but helped to facilitate open dialogue and ensured all OCA participants understood the various capacity areas and the statements they were scoring. In this sense, the OCA scores were based on the perceptions of the people participating in the assessment. In order to be better able to compare OCA scores, it would have been ideal to include the same people who participated in the OCA in the Re-OCA. Unfortunately, this was not possible due to attrition, organizational growth and scheduling challenges. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 28 Figure 3: Average OCA and Re-OCA Scores Another challenge inherent in the process of implementing the tool was that participants tended to over-rate their organizations during the first OCA (baseline). The first OCA was often administered around the time organizations received the REACH award. At this point, many organizations had gone through various external assessment processes administered by Pact Malawi as part of the grant￾making process. Despite reassurances from the facilitators that grant funding was not directly tied to the OCA result, organizations typically linked the opportunity for continued funding to the OCA scores and they tended to over-rate themselves at baseline. 4.3 Pact Malawi’s capacity building interventions Targeted Interventions (TIs): These were tailored activities based on sub-grantees‘ priority areas. Pact Malawi in collaboration with CABUNGO provided a multi-day onsite capacity building activity directly addressing a priority area identified by the partner. At the end of the event a plan was developed for partners to follow up on enacting and implementing appropriate changes based on what was learned through the targeted intervention. Key areas addressed through targeted interventions included: program and service delivery, organizational culture, development of external partnerships, board development, resource mobilization and developing or updating strategic plans. During the key informant interviews partners talked openly on how the targeted interventions helped their organizations. Most partners visited indicated that they had developed a new strategic plan or updated the old one and that they had also developed a fundraising strategy. This was attributed to Pact Malawi‘s capacity building support. The re-OCA results also showed that 15 organizations out of 22 and 11 out of 22 had improved in strategic planning and in organization culture respectively. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 29 Figure 4: OCA and Re-OCA results for strategic planning per partner Source: Pact Malawi Re-OCA report 2011 4.4 Mentoring and coaching in financial management and compliance, monitoring, evaluation, reporting and learning (MERL). Mentoring and coaching was delivered through quarterly support visits to partners by Pact Malawi technical, monitoring and evaluation and finance and grants staff. The sentiments expressed by the vast majority of partners were that quarterly support visits were the most significant aspect of Pact Malawi‘s capacity building support. The majority (85%) of the key informants indicated that they were very satisfied with the support received during quarterly visits and 15% indicated that they were satisfied. One key informant said; ―I found support visits very helpful because the feedback we were receiving as an organization was very constructive. In addition, support visits helped us identify the issues as they came and got assistance to address the issues right away.” (Anonymous) During the evaluation, partners spoke at length about improved financial management processes that they attributed to Pact Malawi‘s capacity building efforts through support visits. Pact Malawi conducted an assessment of partners‘ financial management capacity at baseline with all partners in order to determine the level of managerial and financial risk for each of the potential partners. The MCAT measured administrative and financial management capacity of sub-partners, including accounting procedures, internal controls, budgeting, reporting and auditing and organizational policy environment. The results of assessment showed that all sub-partners would benefit from improved financial management capacity. As Figure 4 illustrates, of the nine partners whose MCAT baseline scores were available, eight demonstrated increases in their MCAT score relative to baseline, while one partner obtained relatively the same score. Notably, changes were more significant for partners whose baseline scores were lower. COPRED, for example, the organization with the lowest baseline score Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 30 out of the sample, achieved the highest percentage change in score, moving from moderately high risk designation at 46% financial and management capacity, to moderately low risk designation at 71%. Also notable are increases in NAPHAM‘s scores from 62% to 85%, and MACRO‘s from 71% to 90%. The increases in scores for SWAM, Matunkha, Lusubilo, FOCUS, and PIH were less dramatic; however, these organizations, as well as AHS, were designated as moderately low risk or low risk at onset, thus the lack of significant change in their MCAT score is not surprising. An abridged version of Pact Malawi‘s MCAT with a sample of nine sub-grantees showed that out of the 9 partners 8 demonstrated increases in their MCAT score relative to baseline (figure 5. Figure 5: MCAT Scores of Selected Partners Risk Level Determination Extremely High 0-20% High 21-35% Moderately High 36-50% Moderate 51-70% Moderately Low 71-85% Low 86-100% Partners also spoke highly of their improved capacity to effectively manage funds which they attributed to regular trainings that Pact Malawi grant and finance staff continuously gave them through the support visits. Results of the Re-OCA also confirmed this finding. The results showed that 12 out of 22 organizations improved in material and financial resources. (See figure 5 below): Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 31 Figure 6: Material and Financial Resources OCA and Re-OCA Scores per Partner Source: Pact Malawi Re-OCA report 2011 In-depth assessment revealed that during the support visits Pact Malawi grants and finance staff could audit program expenditure versus budget, verify expenditure supporting documents and monitor compliance with USAID rules. In the area of monitoring and evaluation, consultations with Pact Malawi sub-grantees revealed that throughout the community REACH program, great emphasis was placed on the importance of having an effective monitoring and evaluation system in place and all the sixteen partners visited indicated that the area they had improved most due to Pact Malawi‘s capacity development support was monitoring and evaluation. This was due to effective support visits and training by Pact Malawi staff. Partners indicated that the data quality assessments led to improved data collection, collation, analysis as well as tracking in their individual organizations. ―Our organization has now a monitoring and evaluation system in place that we use to validate our data.” — Partner NGO leader The results of data risk assessment conducted by Pact Malawi on its partners showed that 21 out of 25 organizations had a high risk level at baseline in 2008 and a second assessment showed that only 4 organizations out of 25 had a high risk level. Third and fourth assessments indicated that none of the organizations had a high risk level, meaning that partners had very high capacity in M&E which implied that they had good M&E systems in place, trained personnel and had good practices in data tracking and usage. 4.5 Multi partner capacity building events These were group trainings offered to various partners based on shared organizational strengthening needs. Partners were asked to participate in at least two out of there of the trainings offered during the year. Training included grants management, monitoring and evaluation, strategic planning, human resources management, governance, leadership and change, advocacy for service providers, gender Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 32 mainstreaming in HIV/AIDS programming, community mobilization, resource mobilization and working with the media and writing success stories. In all the local organisations consulted for this evaluation, officers cited improved human resources processes, improved strategic planning and thinking and enhanced organizational profile as some of the areas that they had recorded significant improvement which they attributed to multi partner capacity building training by Pact Malawi. Partners particularly talked enthusiastically about how the human resource training led to improved human resource systems and most of them indicated that they now have human resource manuals based on the training they received from Pact Malawi. The results of the Re-OCA also showed that most partners recorded improvements in their human resource management and development. Out of the 22 partners whose OCA and Re-OCA results were available, 13 partners indicated improvements. 4.6 Partner review meetings Pact Malawi conducted regular review meetings with its partners which provided a forum for sub￾grantees to improve learning and sharing amongst partners through best approaches and lessons learnt. The meetings also provided a forum to jointly review progress of partners‘ projects, conduct group capacity building, and provide an opportunity for feedback on Pact Malawi‘s performance. Sub-partners‘ perceptions indicated that review meetings were very helpful and promoted sharing knowledge and experiences among partners. This maximized resource utilization and at the same time avoided effort duplication. Sub-partners also complemented how well the review meetings strengthened their relationship and partnership with Pact Malawi through direct and meaningful interaction on a regular basis. During consultations, partners also described improved networking and collaboration with other Pact Malawi partners which they attributed to Pact Malawi‘s regular review meetings. ―Before Pact Malawi’s funding, we had few partners whom we could share knowledge and experiences with, but through the review meetings that Pact Malawi was conducting; now we have more than 20 partners with whom we rely upon to share best practices and lessons.” —Lilongwe based sub-partner official 4.7 External Partnerships and networking Consultations with Pact Malawi sub-grantees revealed that throughout the Community REACH program, great emphasis was placed on the importance of partnerships and networking and ensuring stakeholder involvement in the activities of partner organizations. The results of the Re-OCA also showed that most partners recorded improvements in partnerships. Out of the 22 partners whose OCA and Re-OCA results were available, 16 partners indicated improvements. See figure 6 below: Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 33 Figure 7: External Partnerships and Networking OCA and Re-OCA Scores per Partner Source: Pact Malawi Re-OCA report 2011 In addition to discussing various issues pertaining to programming, Pact Malawi used the partner review meetings to conduct Organizational Network Analysis (ONA) to encourage networking among partners. The tool was administered as well as results presented at the same forum. Discussion around networking during presentation of results encouraged partners to realize the importance of networking. Again the results of the organizational network analysis conducted by Pact Malawi on its partners showed an increase from 19% on baseline to 44%. During the evaluation, partners spoke openly that during Pact Malawi program, they had the opportunity to link with their peers and also to meet government officials, potential funders and members of other NGOs through the various meetings and trainings organized by Pact Malawi. Sub-partners attributed enhanced networking and collaboration with other HIV/AIDS organizations in Malawi to Pact Malawi‘s capacity building initiative. During consultations, partners also discussed networking and collaboration with other Pact Malawi partners as well as non partners and talked about learning innovating strategies through these interactions. For example, NGOs implementing community PMTCT shared how they had learnt to encourage men to take part in ante-natal visits which they learnt from another NGO. Partner-initiated exchange visits also enhanced networking. 4.8 Financial systems and compliance Partner staff received financial management and accounting technical support. Pact Malawi also supported each partner to establish and strengthen the financial management and accounting systems that helped partners meet the USG regulations and compliance requirements. During the evaluation partners indicated that their organisations had improved financial management processes as a result of the mentoring and trainings they received from Pact Malawi‘s capacity building efforts. Sub-grantees‘ perceptions gathered during the evaluation revealed that most partners had formal internal controls governing all financial operations and those operations were fully tracked, supported and reported. A trend was also notable across all sixteen sub-grantees visited that there was clear division of duties in the finance departments and that qualified staff were recruited to manage the finances and assets for Pact Malawi partners. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 34 It was also noted during the evaluation that most partners were able to produce timely and accurate financial reports on monthly basis as per Pact Malawi‘s requirements. It was also evident during the evaluation that most partners had improved in the area of auditing. The pre-award assessment that Pact Malawi conducted in 2007 and 2008 respectively using the Management Capacity assessment tool (MCAT) indicated that many partners were lagging behind in the areas of auditing. However, during the evaluation, the consultants noted improvements in the frequency of external audits the partners had undertaken within Pact Malawi‘s funding period. Although some audits were commissioned by donors as part of the requirements for funding, partners still attributed improved audits as a result of training and mentorship in financial management received from Pact Malawi. Stakeholder consultations at district level largely mirror the findings pertaining to improved financial management processes in Pact Malawi partners. Two of the Pact Malawi partners were awarded grants that they were sub-granting to CBOs in their respective districts. One stakeholder said, “This NGO is a learning centre for all NGO’s and CBOs in this district. It has high capacity in both program and financial management.” (Anonymous) In general, during the time of the evaluation sub-partners showed more initiatives in mobilizing resources. Various sub-partners had started small scale Income Generating Activities (IGAs) in order to diversify their resource base. This was not the trend during the pre-award assessment where all partners had no internal revenue generation activities. However, it was established that partners were finding it difficult to mobilize financial resources to sustain the level of activity that Pact Malawi was supporting. 4.9 Data management systems Sub-partners‘ perceptions through in-depth interviews indicated that there were remarkable improvements in partner organizations in Data Management Systems. One-on-one interviews with key staff in sub-partner organizations indicated that there were high improvements in their monitoring, evaluation and reporting systems which is attributable to Pact Malawi‘s capacity building efforts. Partners who provided feedback on one-on-one interviews also attested to the remarkable improvements partners attained in this area. In fact all the sixteen partners sampled for this evaluation indicated that this was the one in which the partners had improved most. Sub-partners recorded significant results in relation to: Improved adherence to data protocol and data quality criteria Regular practices in data usage in terms of ability to report and validate data accurately Increased number of trained personnel and volunteers in monitoring and evaluation including volunteers Rigorous data collection, analysis and tracking A review of the data risk assessment also attested to the remarkable improvements partners attained in sub-partners‘ monitoring and evaluation capacity. A data risk assessment was conducted with 25 partners in 2008 as baseline to assess the capacity of sub-grantees in data quality assurance and management (figure 8). The capacity areas measured by the assessment were: general M&E systems, data management systems and practices, data quality management and data use for program management and decision making. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 35 Figure 8: Risk Levels of Local Partners by Year Source: Pact Malawi Re-OCA report 2011 Data risk assessment results showed that during the baseline, 21 out of 25 organizations had very poor capacity in M & E which implied that they had poor M&E systems in place, no trained personnel and had no practices in data tracking and usage. The results of the third assessment conducted in 2010 showed that 71 percent of the sub-grantees had made significant improvements in their monitoring and evaluation systems. In-depth assessments also mirrored the findings of the data assessment. Sub-partners indicated that they had clear documentation of monitoring and evaluation procedures and had established systems and tools for data collection, collation and analysis which they now use across their organization. They also had data verification systems in place including internal data quality audits which most say they were not in place before Pact Malawi program. All the sixteen partners visited during the evaluation indicated that they had full time staff in monitoring and evaluation during Pact Malawi program. However, consultants noted that most of the sub-grantees recruited staff in designated areas for example in M&E in order to fulfil Pact Malawi‘s requirement as a donor. During the evaluation, some partners had terminated the employment of their staff in M&E, program and finance departments because they were recruited solely for Pact Malawi program and there were no funds to retain them beyond the Pact Community REACH program. This problem highlights the limitations placed on an organisation that has very high dependence on limited funding sources. This raises a question of sustainability and the ability of sub-partners to move beyond donor compliance-driven change improvements at organizational level. Future program designs should include multiple program phases and rigorous efforts to support partners with technical assistance in resource mobilisation. 4.10 Organizational systems During the evaluation, partners talked about many broad organizational improvements which they attributed to Pact Malawi‘s capacity building initiative. Some partners through a questionnaire gave a comparison of their organizational capacity before and after community REACH program and some of them are in Table 2 below: Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 36 Table 2: Comparison of Sub-Partner Organizational Development Issues Before Pact Intervention (2007) With Pact Intervention (2011) No strategic plan in place Strategic Plans with their operational plans in place Weak procurement planning and implementation Revised organizational policies which include procurement procedures and organizations are able to plan for their procurement in advance and implement accordingly No HIV workplace policy HIV workplace policy in place and operational No volunteer management strategy Volunteer management strategy developed and owned by all relevant stakeholders Board members not knowledgeable on governance and leadership roles Board members underwent governance, leadership and management trainings organized by Pact and HIVOS respectively Board members are more able to assist the senior management team Inadequate advocacy skills Staff and volunteers have skills needed to undertake advocacy activities No fundraising strategy Fundraising strategy On a number of occasions sub-partners spoke about greater involvement of the board of directors and outstanding commitment to the organizations‘ success, mission and vision. They also described their board members providing direction, support and accountability to programmatic leadership and improved participation in major decisions which they attributed to Pact Malawi‘s capacity building. Sub-partners also indicated improvements in the internal culture of participation, empowerment, accountability and transparency present in their organizations that encouraged the professional growth of staff. Specifically, some partners attributed their ability to mobilize resources from new donors as a result of Pact Malawi‘s capacity building support. ―We have accessed other funding as a result of the capacity building by Pact Malawi.” —Executive Director of local NGO (Anonymous) 4.11 Pact Malawi’s implementation of OD support through a Malawian NGO (CABUNGO) The experience of contracting CABUNGO to deliver activities under Pact Malawi‘s capacity building initiative had been largely successful. In general, partners were positive about their experience with CABUNGO. Many partners felt that there was value to having a Malawian organization carry out these interventions (especially the OCA); since the process was done largely in Chichewa, allowing participants to express themselves better. Partners also felt that as a Malawian organization, CABUNGO had a good understanding of the context in which the partners operated and were thus able to tailor the training and direct discussions in ways that were useful for the partners. Additionally, partners praised CABUNGO for their professionalism. Partners felt that CABUNGO‘s facilitation of the OCA and MPCBEs made sessions feel open and encouraged participation. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 37 CABUNGO successfully facilitated institutional capacity assessments that enabled partner organizations to identify their own strengths and challenges, and based on the results helped them undertake measures to enhance their capacity in areas they identified as important. However, most partners felt that the time allocated to do the OCA, the ISP and the follow up was very limited, thus no hands-on assistance was available for them to begin implementing organizational development interventions which they identified through the process (because the follow up – mentoring and targeted interventions – were delayed or considered insufficient). Additionally, an issue that many partners identified was unclear communication regarding dates for interventions; many partners said that CABUNGO changed the dates of training frequently before settling on a final date. This is one of the areas where Pact Malawi needed to provide on-going mentoring and capacity building support to its partners to ensure that follow up OD interventions were implemented. Another comment from multiple partners was that CABUNGO‘s implementation of OD interventions occurred too long after the OCA. Both Pact Malawi and CABUNGO said that one source of delay in the implementation of OD interventions was that neither organization had a clear sense of the resources and time required to carry out the initial work plan. There were contractual issues in play as well, with the initial contract with CABUNGO stipulating that it would use its own resources to carry out activities and be reimbursed. However, as CABUNGO began implementing, it became clear that it was not in a position to cover the costs of training, so it had to postpone the planned activities until contractual issues were resolved. The initial contract, according to CABUNGO, was a learning experience for both parties, and adjustments were required once implementation began. Regardless of the above challenges, most partners still attributed the improvements that had taken place in their organizations to capacity building interventions that Pact and CABUNGO implemented in their organizations. 4.11 Challenges of the capacity building support Inadequate resources Although most partners cited positive changes in their organizations, a number of partners felt that Pact Malawi was under-staffed and were not able to provide as much and as frequent assistance as needed. The sentiment also expressed by many partners was that there were limited financial resources allocated for implementing partners‘ institutional strengthening plans which resulted in some of the issues identified as capacity building priority areas not implemented. The Pact Malawi program did not include this kind of support in its design although the sub-partners made this need known. Future program design should include financial resources support for partner institutional strengthening in addition to technical grants provision. In general, partners felt that more time and resources should have been allocated to the OD process especially for the OCA and ISP, allowing organizations to thoroughly follow up on issues identified. Additionally, a common sentiment expressed was that more time should have been allocated to action planning after issues were identified through the OCA so that there was a clear timeline and delineation of responsibilities for carrying out the identified tasks. “Our organization was asked to source $5000 for implementing ISPs from other sources which we did not manage and this had a negative impact because most of the items in the ISP were not implemented.” (Anonymous) Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 38 Sub-partners‘ perceptions gathered through one-on-one interviews revealed that organization development initiatives started in 2009 which most partners felt that it was an added on activity for Pact Malawi while the initial project implementation and sub-granting process started in 2007. Some partners spoke openly that their proposals to Pact Malawi did not include any budget for Organization Development (OD) activities and implementation of ISPs at an organizational level was a challenge because there was no budget attached. A Pact Malawi key informant verified that there was indeed no separate budget attached to organizational development support to sub-grantees and that Pact Malawi did not have adequate staffing levels to meet the workload that was available and this posed a great challenge to effective implementation of capacity building activities at sub-grantee level. Recommendation Capacity development initiatives require resources in terms of time, material, human and financial. All capacity development initiatives in the future should be attached to budgets in order to avoid the situation where by partners plan for the initiatives but never implement them because of lack of resources. Inadequate human resources and time to assess the impact of the capacity building initiatives Most partners cited positive changes in many capacity areas. During the evaluation it was also noted that Pact Malawi had established a filing system for every partner. In-depth assessment with key informants also revealed that Pact Malawi did not have adequate numbers of field staff to give the time to their partners required for formal processes to assess the impact of each of the capacity building initiative provided to local NGOs. However, partners indicated that sometimes Pact Malawi could make telephone or e-mail follow ups and this was also verified by Pact staff. Recommendation In future programming, Pact should consider increasing the number of monitoring and evaluation staff in order to increase supervision time for tracking progress and impact of capacity building efforts for the partners. Along with this a system should be developed for documenting significant interaction and outcomes of the monitoring visits. This will ensure that key issues discussed e.g. the outcomes of the interaction, the plan of action and expectations and follow up plan lead to agreed actions. This system will enable the capturing of Pact‘s work with partners over time along all capacity areas. Exclusion of OD component in quarterly support mentoring visits Partners‘ perceptions gathered through in-depth assessments revealed that it was Pact Malawi‘s finance, program and monitoring and evaluation staff that undertook field support visits. Pact Malawi did not conduct separate OD meetings with partner organizations at an organizational level to discuss or follow up on OD issues. The exclusion of the OD staff in the support visits was also an indication that OD support was viewed as a less important aspect of the program. A comment from a Blantyre based partner was ―no organization development specialist has ever visited us from Pact Malawi to give us the organization development support that we needed.‖ Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 39 Recommendation In future programming, Pact should consider elevating the OD aspect by including it in the quarterly support visits and reinforcing it by tracking OD progress in quarterly reporting. Lessons learnt on capacity development support to Local NGOs 1. Pact Malawi‘s direct mentorship of its local partner NGOs and CSOs in organizational development effectively strengthened their capacity to develop, adopt and implement improved systems within their organizations and was an excellent supplement to formal training. 2. Local Pact Malawi partner NGOs that had robust financial systems more potential of attracting local and international donors. 3. Through Pact Malawi‘s program approach it was clearly demonstrated that networking and collaboration among local NGOs was a key for success in terms of sharing of strategies and best practices that allowed service providers to acquire and enhance their service delivery skills, use appropriate and relevant techniques and avoid duplication of interventions. 4. Pact Malawi‘s establishment of a monitoring and evaluation department enabled an organization to keep better track of program data. It also allowed for this kind of data to be more easily accessed and utilized. It is essential that monitoring data be used to inform program decision making. 5. Pact capacity development support that combined both technical as well as organizational development proved effective in improving local organisations. OD interventions supported partners to improve their organizational systems, leadership and management, strategic planning, governance and monitoring and evaluation among others. These were areas of great need at the beginning of the program. The technical aspects improved their service delivery skills. By combining the two, the interventions became richer and more effective. 6. Formation of data quality teams by Pact Malawi partners which constituted cross functional teams made up of staff, local leaders and other stakeholders with different levels of responsibility made the results and achievements of partners rich because this promoted shared ownership and responsibility for ensuring that the compilation and reporting of results was accurate and well documented. 7. Pact Malawi Community REACH program approach with its partners demonstrated that capacity building of local communities is essential for project sustainability and ownership and increased service uptake. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 40 SECTION FIVE 5. FINDINGS ON PACT COMMUNITY REACH PROGRAM TECHNICAL SUPPORT INTERVENTIONS Pact Malawi provided its implementing partners with technical assistance in various aspects of HIV/AIDS services including monitoring and evaluation. Pact Malawi program approaches were employed depending on the specific needs of the program area as well as the level of technical understanding of the sub partners. The needs for technical assistance were identified through Pre – award meetings, multi-partners review meetings, feedback provided during reporting, results from OCA process, changes in the PEPFAR indicators and gaps identified during the regular monitoring visits by Pact Malawi staff. Technical approaches used by Pact Malawi were effective in supporting its local partners to improve their capacities to deliver and incorporate services in their programs. The level of partner programming quality and reporting demonstrated that Pact Malawi‘s support led to the improvement of sub-partners‘ technical capabilities and capacities for achieving desired and planned health outcomes in the communities where they had interventions. Through in depth interviews with partners it was evident that the partners benefitted very much from the technical assistance, for example, the programming knowledge and skills of partners improved and that helped organizations to yield desirable results. In addition to this, partners improved their data management skills through an effective monitoring and evaluation system which Pact Malawi helped them to roll out. It was pleasing to note that the stakeholders including the DHO in some districts benefited from the DQAs that they started aligning their systems based on the one Pact Malawi introduced. Other benefits included formation of linkages, partnerships and synergies with other partners with similar programs. These partnerships ensured sustainability of some of the activities the sub partners were implementing under the community REACH program. This section of the report highlights key findings of the evaluation which was conducted among 16 of the 23 Pact Malawi partners. In this section, emphasis will be placed on the broad spectrum of technical program activities in different areas as presented below. 5.1 Intervention 1: HIV Prevention In order to yield desirable results on the technical program areas, Pact Malawi integrated the technical assistance aspect. The partners focused on HIV prevention for the general population through abstinence, being faithful, and condom use, PMTCT, Community PMTCT, prevention for most at risk populations, Prevention with the Positives and Post Exposure Prophylaxis (PEP). The prevention model which Pact implemented appears to be appropriate and has contributed to the National response of HIV and AIDS. Pact Malawi has provided technical assistance in this respect which included support visits, coaching, trainings to partners and mentoring. The technical assistance resulted in improved delivery as evidenced by overachievements of the indicators as well as replicating of the approaches by some partners who indicated during partners‘ interviews that Pact‘s approach was appropriate and feasible. In order to achieve the intended results the technical assistance was extended to volunteers, stakeholders, peer educators, counsellors and employed staff from sub-grantees. The report details the key findings of each prevention activity to demonstrate how the technical assistance has contributed towards the attainment of the prevention outcomes as described below: Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 41 i. The Abstinence, Be Faithful and Condom (ABC) Approach (Prevention) This approach was mainly used to guide the delivery of HIV prevention messages that included abstinence, being faithful and condom use. The findings indicated that the approach was very effective in raising HIV/AIDS awareness and most importantly, the changes in the risky behaviours of the population. The effectiveness was attributed to adoption and use of various methods in reaching out to the people and it was worth noting that the messages were age-specific and putting more emphasis on the A and B especially to the youths. The key prevention messages delivered focused on the importance of reducing multiple concurrent sexual partners, communication, life skills and correct and consistent use of condoms. The program overachieved its targets, indicating how committed the staff were in implementing the program. During the course of the implementation, PEPFAR introduced the next generation indicators which proved to be superior in terms of measuring behaviour change as they encourage one–on–one discussions with peer educators and use of small groups. The next generation indicators which promoted reaching an individual thrice with prevention messages encouraged beneficiaries to be more open with peer educators and as a result encouraged people to seek other HIV and AIDS related services. Major Achievements Peer to peer education sessions proved to be very effective in behaviour change communication. The approach used one-to-one and small groups of maximum 25 people. Its success lied in the fact that it enabled participants to be active and participate actively in the sessions unlike the mass campaigns. As a result of a small group, participants were afforded an opportunity to ask questions or seek clarity and even engage the peer educators in one-on-one discussion. “I find the work of peer educators meaningful as I am shy and cannot ask questions in public but through the one-on-one discussions I had with the peer educator I was able to ask all HIV related questions that bother me and finally I went for HIV test and the outcome was negative. Since then, I have ended my extramarital relations and I encouraged my wife also to go for HIV test. We are now a happy and loving family and I credit all this to the peer educator who helped me.”—FGD participant at NASO Another achievement was the record-keeping aspect which volunteers involved in peer education demonstrated in all sites that were visited. Volunteers had records indicating the names of people that they had reached out to and they all demonstrated a superior understanding of the three sessions that were targeted at an individual for him or her to be counted. Partners who implemented this component also organised open gatherings or campaigns with a wide range of activities which included sports, drama and poetry that proved popular. Apart from conveying the ABC messages, the open gatherings kept the youths busy and therefore provided an environment where youths had little time to engage in risky sexual behaviours. HIV Prevention Targeted and Achieved Indicators Indicator FY 2007 to FY 2009 Target Achieved The number of individuals reached through community outreach that promoted HIV and AIDS prevention through abstinence and/or being faithful 426,525 999,745 Number of individuals reached through community outreach that promoted HIV/AIDS prevention through abstinence 255, 255 499,747 Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 42 Indicator FY 2010 to FY 2011 Target Achieved Number of the intended target population reached with individual and/or small group level interventions that were based on evidence and/or met the minimum standards 250, 476 427, 046 Number of the targeted population reached with individual and/or small group level preventive interventions that were primarily focused on abstinence and/or being faithful, and were based on evidence 208,089 168,746 The overachievement in targets was attributed to intensive education to partners and volunteers on new prevention approach including development of the minimum package coupled with frequent data quality reviews. Additionally, Community REACH Program had contributed greatly to the national response and the program was effective in scaling up prevention outreach to the beneficiaries and the prevention model implemented by Pact Malawi was appropriate, feasible and replicable. Challenges Retention of volunteers was a problem in some areas as most of them migrated to urban areas or even went beyond the borders to look for jobs and in some cases the single volunteers were forced to relocate once they got married. Other volunteers stopped because they thought they were not adequately motivated. Replacing them was difficult especially providing training to them because of the huge unbudgeted financial outlays associated with such training activities. It appeared among the partners, despite numerous efforts by Pact Malawi, that there was no consensus regarding volunteer motivation. Some partners had good strategies in providing incentives that included T-Shirts, Bicycles, Umbrellas, carrying bags and Identity Cards and many more. The issue could have been adequately addressed by establishing minimum packages for volunteer incentives for motivations. ii. Most At Risk populations (MARPS) The major strength of the program was its ability to implement interventions tailor made to suit the specific needs of the populations. The MARP included Commercial Sex Workers (CSW), fishermen, plantation workers and other vulnerable populations such as men in uniform, construction workers; bicycle taxis boys, ritual cleansers, teachers and others. During the assignment, policemen, construction workers and teachers were interviewed. The findings illustrated that these MARPS were reached out with HIV prevention messages and also condoms were made available to them. During Focus Group Discussions (FGDs) with the beneficiaries, it was evident that the intervention was just newly introduced and had not taken off yet compared to other prevention interventions. Despite that, it came very clear that it targeted hard-to-reach groups like police officers and construction workers who by nature of their work put them at a higher risk due mobility and staying away from families for prolonged periods. Major Achievement The program managed to create risk perception among the MARPS and behaviour change in terms of reducing multiple and concurrent partners seemed to be taking roots in all categories of people interviewed. It was clear that those who could not abstain were reported to be using condoms consistently and this was evident amongst the construction workers. From FY 2007 to FY 2009, number of condom service outlets were 1,184 from a target of 818 and 871,138 individuals were reached through community outreach that promoted HIV and AIDS prevention through other Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 43 behaviour change beyond abstinence and/or being faithful against the planned 545,900. It was also interesting to note that from FY2010 to FY2011 the number of condom outlets increased from the planned 495 to 693 and 24,096 MARPS were reached versus the target of 15,292. Challenges Few police officers were trained as peer educators and many police officers did not have adequate and correct HIV and AIDS information. There was a need to train and support more peer educators within the police who in turn could conduct peer to peer education sessions with fellow officers. In case of police officers, it was difficult to make any conclusions on behaviour change because the program period was short and also the prevention training given was not adequate as it did not reach all police officers. The supply of condoms to the MARPS was intermittent and more male condoms were distributed than the female ones. Only three officers belonging to Victim Support Unit were trained in PEP and this made it difficult for other officers to respond swiftly to victims of rape by referring them quickly to the hospital for PEP. In addition to that, PEP training excluded Traffic Police Officers who often times deal with road accident victims and are exposed to body fluids like blood. There were no peer educators within the construction workers group and this was a missed opportunity which could have been vital in disseminating HIV prevention messages. iii. Prevention of Mother To Child Transmission (PMTCT) Malawi has been scaling up its PMTCT services throughout the country and Pact Malawi continued to support four Faith Based Organizations (FBOs) namely Ekwendeni, Malamulo, AHS and Nkhoma in providing PMTCT services. These FBOs manage 22 health facilities of which all were supported by Pact Malawi through mentorship and training. The component registered some successes. It is also important to note that the PMTCT component was implemented through a two prong approach namely hospital based PMTCT and community based PMTCT. The hospital based PMTCT focused mainly on the biomedical services where pregnant mothers were provided with ARV prophylaxis to prevent the mother to child transmission of HIV. The pregnant women were provided with the single dose nevirapine or women were provided with a combination of two ARVs and in some cases the Highly Active Antiretroviral Therapy (HAART) was provided. All women regardless of the type of ARV prophylaxis were also provided with cotrimoxazole during the same period. The outcome of the ARV prophylaxis proved effective, for example, 97% of children that were born to positive mothers, tested negative in 2011 an improvement from 64% registered in 2008. A careful analysis still unveiled that facilities where combined ARV therapy was administered registered a lower MTCT rate than where monotherapy (single dose niverapine) was administered. Depending on this analysis, Pact Malawi advocated for the scaling up of the use of combined therapy on the health facilities they supported and this contributed towards impressive test results for the exposed infants. All exposed infants received cotrimoxazole prophylaxis from six weeks of birth until tested HIV negative. Realizing the fact that positive pregnant women spend most of their times outside the health facilities, Pact Malawi developed a strategy for community PMTCT, with approval of USAID and Ministry of Health, to provide support and care outside the health facility to HIV infected mothers, their exposed or HIV positive infants and families to satisfy their unmet needs. At the same time the initiative promoted access of PMTCT services in the health facilities from the time the woman enrolled into care until the child‘s sero status was confirmed HIV positive or negative and linked to Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 44 appropriate care. Pact Malawi, through its partners utilized volunteers to conduct home visits to households to identify beneficiaries; provided PMTCT information to reproductive age couples. For HIV positive clients, volunteers provided all information regarding PMTCT services and checked on progress until the child was tested for HIV. Major Achievements Pact Malawi supported 22 centres (FY2007 to FY2011) providing a minimum package of PMTCT services according to both national and international standards against the target of 20 centres and during the same period 72,590 women underwent HIV testing and counselling within the context of PMTCT against a target of 71,618. Of the women tested 4,723 (6.5%) were positive and received antiretroviral prophylaxis against a target of 3,901. Over the five year period of Pact Malawi implementing the PMTCT intervention, remarkable results were achieved as presented in table 3 below: Table 3: Impact of Pact Malawi’s PMTCT intervention Year Number tested Number Negative Percentage negative 2010 to 2011 590 573 97% 2009 to 2010 443 383 86.5% 2007 to 2008 44 28 64% Total 1,077 984 91.4% The results showed that there was a downward trend in the number of babies born HIV positive from 2007 to 2011. A more significant improvement was registered from October 2009 to 2011. This was the period when the next generation indicators were introduced with emphasis on provision of a combined regimen to PMTCT clients which was found to be more effective that the single dose administered from April 2007 to September 2009. The results were enhanced by follow up and technical support intensified from October 2009 to the end of the project in December 2011. Another achievement was the strengthening of the MOH guidelines on the integration of HTC in the Maternal and Child Health services. This improved uptake of PMTCT services in all the four FBOs which Pact Malawi worked with. The integration of HTC services meant that HIV positive women who came for ANC, labour and delivery services as well as family planning services were easily identified and recruited into PMTCT services. Pact Malawi and its partners developed a rigorous system of tracking mothers and infants which involved community volunteers and the hospitals coupled with a good referral system as a result this made it extremely difficult for defaulters to go unnoticed. The volunteers had referral forms which they used to refer clients to hospitals. This practice ensured that clients including possible defaulters were traced for follow –up visits. From the FGDs it became apparent that most communities abandoned going to TBAs and opted for hospitals solely due to PMTCT services. In turn, more women attended ANC services and thereby reducing preventable infant and maternal deaths. In addition women through what they experienced in their day to day lives came to realize the importance of PMTCT. During the FGD at Tovwirane increasing numbers of women went for HIV tests. Those found positive would then go for PMTCT prophylaxis. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 45 “I tested HIV positive and disclosed to the nurses during my ANC visits, I was put on special medication and delivered a healthy baby who tested negative and he is well over two years.”—Tovwirane FGD participant Through community mobilization, Pact Malawi and partners created demand for HTC, ART and PMTCT as more and more people were willing to go for HTC so that they could benefit from other programs and this was attributed to the introduction of community PMTCT concept. Partners established very good working relationship with DHOs, other private hospitals like Dwangwa where these services were provided. Challenges Despite Pact Malawi being successful in creating demand for PMTCT services, its efforts may be dwarfed if the chronic shortage of test kits continues unabated. More women reported being turned away due to shortage of test kits at almost all health facilities. The stock out of test kits had a discouraging effect on women who went for ANC services. The emphasis of PMTCT services had been to women who only test positive during their ANC visits. Those who tested negative were not properly monitored. Recognising the fact that during the first visit the women might have been on a window period, Pact Malawi partners should have advocated for testing women in all trimesters. Stock out of PMTCT drugs was reported in most centres. This disrupted the efforts made in sensitizing the women on the importance of knowing one‘s status so as to access PMTCT services. There were delays in getting infant PCR results from central laboratories. Most beneficiaries reported on this as a way in which they needed to see improvements in the PMTCT services. PMTCT Individual success story Mrs. Wound (nickname) is young woman aged 29 years old. She got married to a husband who died after developing chronic diarrhoea for three good months without stopping in 2007. He was tested HIV + in 2005 and he did not tell his status to the wife. At the time of his death, the wife was by then four months pregnant. She was also having the problem of vomiting anything she took because of her pregnancy and lost weight drastically When she came to Nkhoma hospital she was tested HIV positive and counselled for PMTCT services of which she understood the concept so easily since she went as far as JCE at school. She accepted the situation in which she was and started ANC services immediately. She was given Nevirapine tablets to take when the labour starts. She gave birth to a baby boy in 2008. Sometime in June, 2008 she gave blood for CD4 count which came out to be 24cell/mm3 and started ART immediately. Her child called Manuel was tested at 1 year for HIV and was found to be negative and the mother continued to give him cotrimoxazole preventive therapy with hope and courage. iv. Post Exposure Prophylaxis This was provided by Nurses and Midwives Council of Malawi (NMCM). Major Achievement There were 294 health facilities (out of a target of 250) with HIV post exposure prophylaxis available and 406 patients or clients were provided with PEP services against the planned 300. Challenges The departure of the focal person who had oversight of the PEP program at NMCM made the program unsustainable. This was evident during the in-depth discussion with key personnel at NMCM who utmost demonstrated little knowledge of how the program was being implemented. This was further Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 46 illustrated by their inability to know the capacity development and technical initiatives Pact Malawi rendered throughout the entire period of the program. Institutional memory was lost due to staff changes at NMCM. There was no standardised way of recording the beneficiaries amongst all the hospitals that implemented PEP under NMCM. Considering how PEPFAR emphasizes the importance of data management, the program should have diagnosed and eventually developed the standard tools to capture the information. The program registered a high default rate as it was reported that less than 2% of the beneficiaries reported back for subsequent follow – ups as required by the PEP protocols. There was little investment in training to the extent that few providers and other key stakeholders, like the police, were just given one day training. The content of the training was easy to digest for those people in ART clinics. However, police officers indicated that they needed more time for the training as their knowledge of subject was understandably limited. v. Prevention with Positives Pact Malawi partners were implementing community Prevention with Positives (PwP) by combining the approaches of clinical assessments and interventions with information sharing. This was piloted by partners who were also implementing palliative care including NASO, Nkhoma, MAICC, Ekwendeni, SAT (LIHASO), COPRED, Malamulo and Tovwirane. PwP proved to be a very good model of prevention among infected people. The program was also consistent with the principles of Pre –ART care and treatment. It presented an opportunity to impart HIV prevention messages. The initiative had just been introduced and a desk review indicated that it did meet its target and if scaled up it had the potential of reducing re-infection and help enhance behaviour change among the PLHIV. Major Achievements Over the 2 year period of implementing this component of the program, Pact Malawi reached 33,186 People Living with HIV/AIDS (PLHIV) with a minimum package of prevention with PLHIV (PwP) interventions against a target of 24,750. Provision of risk reduction information among the PLHIV coupled with the distribution of condoms and lubricants, prevented the PLHIV from re – infection and infecting others. Challenges There was a reported inadequate supply of condoms and IEC materials to the clients from the DHO to the intended groups. This greatly undermined the effort of CSOs who had invested in training peer educators and conducting training. Most partners implementing this component acknowledged that they did not have the required knowledge to implement prevention with positives. The technical assistance provided by Pact Malawi was inadequate suggesting that even Pact Malawi did not have the capacity to impart knowledge to the partners to successfully roll out the prevention with positives concept. 5.2 Intervention II: HIV Testing and Counselling HIV Testing and counselling (HTC) is the heart of HIV/AIDS services. It is the entry point for prevention, care and treatment. The test enhances an individual‘s risk perception to the extent that when one knows his/her status avoids infection or prevents further re-infection. Pact Malawi Community REACH program worked with CSOs and MOH to set up and in certain cases to strengthen the already existing HTC centres. These HTC centres included static, mobile and outreach sites. It is also important to highlight that Pact Malawi and its partners, in order to reach out to as Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 47 many people as possible through its technical assistance, introduced door to door counselling and testing services in some areas. Other innovations by Pact Malawi partners under HTC included the following: a. Male championship that encouraged couple counselling and in the process discordant couples were identified. b. Moonlight candles by MACRO to provide HTC services to mobile populations such as truck drivers and commercial sex workers during the night. Major Achievements Over the 5 year period, Pact Malawi supported a total of 441 (against target of 324) centres providing HTC and these collectively reached 850,751 individuals that received testing and counselling services for HIV and received their test results exceeding its planned target of 654,914. Technical Assistance: Pact Malawi provided technical assistance through training, mentorship of counsellors thereby ensuring quality of HTC provided. Because of the technical assistance rendered, the counsellors were highly motivated and committed to providing HTC services despite the limited resources at their disposal. Increased HTC uptake and improved service: With new HTC sites and more trained counsellors, uptake of HTC services increased rapidly. Most HTC sites were able to provide HTC services. The increase in demand for HTC services was further boosted by the implementation of other HIV services like PMTCT, ART where HTC was the entry point. Due to the availability of care and treatment the uptake of HTC seemed to be on the increase. Supervision, Monitoring and Reporting: Pact Malawi made supervisory visits to provide further technical assistance to all partners providing the HTC services. Ekwendeni Hospital staff regarded supportive supervision as extremely beneficial and it was pleasing to note that the HTC visited had an HTC registration for recording of clients. Challenges Test kit shortages and stock outs: Lack of test kits was evident in all areas visited by the evaluation mission and this was causing service interruption in many of the visited areas. If not reversed then the gains that have been created in other programs like PMTCT are going to falter. The problem is compounded by logistical bottlenecks at both regional and national levels. Distance: The distance that many clients must travel to reach their health centres for VCT can be prohibitive. Even though there were mobile sites (door-to-door) not all areas benefitted from them. This brought out an issue about access not only to HTC but also other services such as ART and PMTCT. Quality control: All partners follow the national guidelines on HTC. These HTC guidelines stipulate quality issues around counselling and testing and also stipulate standards for disposal of test kits and other medical materials. Partners procured supplies needed for provider protection and safe disposal of needles, and other contaminated waste. Incinerators were utilized to dispose of hazardous waste. All waste accumulated from outreach and mobile services was stored in appropriate containers and disposed of at their main centres to avoid leaving contaminated waste in the communities. Partners conducted weekly quality control checks by testing HIV-positive and negative samples. All partners received quality control checks by their respective DHOs. Random samples were taken and sent to the central reference laboratory at CHSU for external quality control. Supervisory visits were also conducted to all sites and supervisor observation counselling was conducted. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 48 HTC Case Study: The discordant couple Discordant HIV status in a couple is often shocking for both partners. For the Kayedzeka family, that was exactly the case. The husband tested HIV positive while the wife tested negative. The results marked the beginning of a new life and also gave them hope as they wanted to have a child. ―After being sick on and off, I decided to take an HIV test in 2007 and was found to be HIV positive. Yes it was disheartening, but then I gathered myself and decided that I’d be in the forefront to educate my colleagues and encourage them to get tested as well,‖ explains Lackson Kayedzeka. Lackson is a beneficiary of home based care provided by the Nkhotakota AIDS Support Organization (NASO). The couple went to the Alinafe Hospital where Lackson has been receiving treatment, and they were advised on the precautions to take to have a child who did not have the HIV virus. They now have a beautiful two months old daughter, and the wife has been able to maintain her negative status. Lackson is a role model in his Mwadzama village that is in the Central Region of Malawi, Nkhotakota district, as he normally educates people about the HIV testing and counseling. Whenever there is some work with the community regarding issues of HIV/AIDS, he has always been at the forefront. Now that the discordant Kayedzeka family is not shocked or surprised anymore, all they are trying to do is assure everybody that being HIV positive is not the end of the world. Intervention III: Palliative Care/Home Based Care (PC/HBC) Pact Malawi focused on promoting improved understanding of and technical expertise in, palliative care amongst its partners and there were significant changes in performance on PC/HBC among the partners and community care providers. Pact Malawi, together with its partners, trained PC/HBC volunteers in the areas of hygiene and home nursing. This established a supportive environment within the communities that provided the following services; counselling and support, nutrition, promotion of safe sexual practices, family planning information, and referrals to medical care services. Volunteers also provided other support services like income generating activities, farming, and psychological support for the issues associated with terminal illness. Thirteen out of 24 partners namely: PIH, COPRED, Ekwendeni, Lighthouse, Lusubilo, MAICC, Malamulo, Matunkha, NAPHAM, NASO, Nkhoma, and Tovwirane provided palliative care services. PACAM was not a service provider but supported a network of palliative care providers to effectively provide services by taking a lead in the development of National Palliative Care Training Manual, conducting orientations and training of service providers of palliative care. It was pleasing to note the strides the partner organizations made towards the provision of Palliative Care/ Home Based Care (PC/HBC). One notable step was the establishment of linkages for referrals which is two-way, from community to hospital and vice versa and also records of the clients were established which helped to track the clients and this made follow-ups easier. PC/HBC was delivered by volunteers and supervised by nurses from the DHOs to individuals and families who often live in social isolation. Major Achievements Pact Malawi supported 162 service outlets providing HIV-related palliative care exceeding its plan of 92 service outlets and 85,000 eligible adults were provided with a minimum of one care service. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 49 Training of PC/HBC volunteers: HBC volunteers provided home care visits to beneficiaries on a regular basis with nurse supervision and oversight as needed for basic nursing and social support. Volunteers and supervisors met to review client care. The volunteers had a home care kit that supports the provision of basic hygiene and nursing activities. The content of HBC kits was standardized by the MOH. Furthermore, volunteers trained family members in PLHIV care. Additional supplies were available to the nurse supervisors or at nearby hospitals to address acute and chronic medical problems, including access to additional analgesics and antibiotics. Reduction of stigma: Self-stigma and fear of discrimination created significant barriers for individuals and families in accessing and/or using HIV care services, including HBC. Pact Malawi HIV/AIDS program fought stigma and discrimination. The presence of HBC services in communities made evident the benefits of using HIV care services through the improved health conditions of beneficiaries who to some extent society condemned them to death. It was evident that this outcome fostered an increased demand for HTC. Initiation of self-help activities: Because of the increasing nutritional demands for most HBC clients, most volunteers were involved in self help initiatives like winter cropping, keeping small ruminants which when they harvested they distributed to HBC clients or used the proceeds from sales to buy toiletries which they used when nursing the clients. PC/HBC manual: Pact Malawi facilitated the creation of the standardised training manual which was used to train volunteers. The training was structured in specific modules addressing service areas that were based on didactic and interactive sessions, problem-based learning with case studies, and other adult learning methods. Apart from the manual, Pact Malawi also developed a minimum package on HBC. The minimum package guided partners in provision of palliative care. The minimum package improved the quality of services rendered to clients. Before this the partners had problems in understanding both the national HBC guidelines and the PEPFAR guidelines. The partners were orientated on the minimum package guidelines and also trained their volunteers on the same. This in turn improved the quality of work on the ground. Challenges Stock outs of HBC Kits: Most areas visited reported stock out of drugs and other supplies. The DHOs, where they got their supplies, did not stock enough drugs. This was greatly compromising their work and in some cases putting the volunteers at risk as they did not use protective clothes and gloves when nursing the clients. Even though the problem had been there for some time but at least gloves were made available easily which is not the case now. Financial and other burdens to volunteers: Volunteers devoted substantial time to their HBC activities and provided a tremendous service. Most volunteers were low-income women with already￾heavy family and other responsibilities. The allowance that most programs provided did not even cover their basic expenses, such as transport to and from beneficiaries‘ homes, to and from training meetings, and to and from hospitals and health centres. In some cases despite the long distances they travelled, the volunteers were not provided with push bikes, umbrellas or other items to make their work easier. 5.3 Intervention IV: Orphans and other Vulnerable Children (OVC) Care and Support The OVC program was implemented while following the national guidelines on OVC developed by the line ministry. Partners based their service provision on the six-plus-one core service as stipulated by the PEPFAR guidelines. The support and care provided were age specific which came in the form of education support which promoted access to Community Based Care Centres (CBCCs) and also provision of scholastic support to the secondary school going age. There was also provision of vocational skills to the out-of-school youths. To enhance the quality of work, Pact Malawi introduced the use of Child Status Index (CSI) tool which assesses the needs of the OVCs amongst its partners. It followed the six core areas of OVC care: health, nutrition, shelter, protection, education and Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 50 psychosocial wellbeing. These six core areas were further divided into domains with two on each core area, making 12 domains in total. Major Achievements Since the inception of the Community REACH Program, Pact Malawi served 177,400 OVCs against the planned target of 125,416 between FY2007 to FY2011 and during the period also 1,174 OVC care givers were trained. The program promoted early childhood development in most areas through the increased number of CBCCs that were established. The children that passed through the CBCC did much better than their counterparts. The provision of vocational skills proved to create self employment jobs to some youths and in some cases some were employed by reputable companies. The youths were able to pay fees for their siblings as well. There was strong evidence in collaboration with other partners, for example, Land O‘ Lakes collaborated with MAICC supported CBCCs and provided milk to OVC program. This is improving the nutritional uptake of children at CBCCs. MAICC also received VitaMeal from Feed the Children which they also distributed to CBCCs they support. The distribution of mosquito nets to the OVCs by the DHO in Rumphi resulted in the reduction of malaria incidences in many households. Child Parliament was also strengthened at NACC and Matunkha where OVCs openly discussed issues affecting their lives. The other successful dimension of the program was the children corner where OVCs discussed their challenges amongst themselves and found solutions. It was also a forum where they got psychosocial support. Apart from the support they got from Pact Malawi, the OVCs, through other stakeholders especially at Matunkha in Rumphi, also generated support from the Department of social welfare, Ministry of Agriculture and received some items such as Likuni Phala, kitchen ware, balls which they used at CBCCs and other primary schools. Pact Malawi supported the government of Malawi‘s initiative in piloting national OVC standards with the NACC. Key innovations during the roll out included baseline assessments, formation of QI multidisciplinary teams and use of data to lobby for more teachers to be deployed in NACC area. The baseline revealed that there were fewer teachers than were needed in some schools. And upon presenting the data to the district education officials, NACC was supported with 24 more teachers. NACC utilised Child Status Index (CSI) to monitor OVC outcomes. The CSI tool helped volunteers to track the delivery of OVC services. The volunteers admitted that it was easier to understand and use to the extent that it helped in assessing the needs of OVCs and addressing them. Challenges The provision of school fees was not properly designed as some students dropped out of school mid way after the program had phased out. No strategies were put in place to hand such kids to social welfare department. Stakeholders As earlier mentioned, there were many actors who contributed to the success of the Community REACH Program. In order to establish the success of the program, evaluators interviewed some of the key stakeholders nationally and locally (at the district level) to solicit their views on the capacity development, technical assistance and sub-granting. It was very clear that Pact Malawi and its Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 51 partners collaborated well with different stakeholders in various HIV and AIDS thematic areas both nationally and locally. The general consensus of all partners interviewed was that the community REACH program was successful as it contributed greatly to HIV/AIDS work and that it transformed the lives of many vulnerable groups and had tremendously improved the quality of life of the orphans and chronically ill patients. The collaborations reduced duplication of efforts and that through networking; stakeholders shared information, mentored each other and shared resources at the district level thereby maximizing the benefits and impact. 5.4 LESSONS LEARNT The technical assistance rendered especially on data verification was very central to the whole community REACH Program. Data quality assessment sessions minimized errors and errors identified were corrected. Door to door HTC service was key to the success of other HIV services. As a result of the initiative many people were tested and the demand for other services for example PMTCT, ART increased. Community based vocational skills through engaging local artisans as trainers proved to be economical and socially viable because orphans and vulnerable children were not alienated from their communities. Community mobilization improved dissemination of information and this helped in HIV reduction, young people intending to get married, coming forward for testing and counselling before wedding ceremony after knowing the importance of one‘s HIV sero-status so that they followed the proper way of living, positively or negatively. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 52 SECTION SIX 6. FINDINGS ON PACT COMMUNITY REACH PROGRAM GRANTS MANAGEMENT AND ADMINISTRATION 6.1 The extent to which the Pact Community REACH program achieved its objectives in relation to provision of sub-granting to local NGOs In the five years of existence, Pact Community REACH program awarded 38 grants totalling some $10 million to a cross section of local organizations ranging from faith based to private sector in the 19 out of 28 districts of Malawi, thus, covering all the three regions of the country. As a result of utilising these grants, Pact Malawi increased local organizations‘ access to financial resources, thus, enabled them to implement programs which addressed specific HIV/AIDs problems in communities. This increased access further assisted local organizations to integrate HIV/AIDS into primary health care system. Sub-grantees also successfully developed linkages with other local organizations and government departments working closely while ensuring that there was no duplication of activities. Pact Malawi used a useful and innovative approach in administering and managing grants given to its partners. The following steps and actions that Pact Malawi undertook represent best and innovative practice: 1. Internal Control Systems: • Preparing policies and procedures before issuing grants • Consolidating information systems to assist in managing grants • Providing grant management training to staff and grantees • Coordinating programs with similar goals and purposes 2. Performance Measures • Linking activities with program goals • Working with grantees to develop performance measures 3. Pre-Award Process • Assessing applicant capability to account for funds • Competing grants to facilitate accountability • Preparing work plans to provide framework for grant accountability • Including clear terms and conditions in grant award documents 4. Managing Performance • Monitoring the financial status of grants • Ensuring results through performance monitoring • Using audits to provide valuable information about grantees • Monitoring sub-recipients as a critical element of grant success 5. Assessing and Using Results • Providing evidence of program success • Identifying ways to improve program performance These promising practices used by Pact Malawi demonstrated opportunities for improving grant accountability. The grants were an important tool used by Pact to achieve its Community REACH program goals while working with 23 local organisations as implementing partners of HIV and Aids interventions and activities in communities across Malawi. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 53 Opportunities for improvement in grant accountability by Pact Malawi and its partners existed throughout the grant process, as shown in the 5 steps below. Prior to awarding grants, it was important for grantees to have internal control systems and performance measures to facilitate grant management. The grantees then needed an effective pre-award process, a process for managing performance once grants were awarded, and the ability to assess grant results and used those results when awarding subsequent grants. Sustainability was one of the components that sub-grantees attached to their program activities. The sub-grants not only empowered sub-grantees but also communities they worked with. The essence of this sustainability was that it used local knowledge in the communities by recruiting volunteers and using and recognising village heads as ‗gate keepers‘. This greatly helped in the continuity of activities such as OVCs, HBCs, peer education, male to male and many others. Many stakeholders and focus group participants had the assurance of the continuity of activities. However, the prospects of sub-grantees sustaining the level of activity without additional funding from other sources than Pact Malawi remained weak. Sub-grants had one hidden effect on the community which was enhanced social relationships (social capital). It was evident from the discussions that there was some social cohesion among those living with HIV/AIDS, volunteers and communities at large. Participants spoke of community contributions in cash or kind to OVC Centres when food items ran out. Support groups also spoke of raising money through a communal income generating activity for their own health emergencies. The money would help transport someone to hospital and buy essential needs easing the family‘s need for transport. 6.2 Achievements of Program’s Objectives The achievement of project‘s objective on sub-granting was attributed to several factors: Firstly, sub-granting had strong alignment with needs of partners and that of beneficiaries in communities. A review of organizations‘ documents showed a history of HIV/AIDS activities in their Implementation Plans an indication that the grant award was a much more welcome opportunity to them to enhance their financial muscle in the implementation of HIV/AIDS program. The sub￾granting therefore was demand/need driven thus was met with organizations‘ enthusiasm and desire to implement activities they knew and were skilful in; yet had financial limitations to implement them effectively. Secondly, the sub-grants supported interventions that were one of government‘s agenda and were prioritised in its set of priorities. As a result, the sub-granting had links with stakeholders at national, district and community level. The presence of high level officials from Office of the President and Cabinet at the Grant Award ceremonies in October, 2007 and September, 2008 and the participation of sub-grantees in national technical working groups was a strong indication of government support to Pact Malawi Community REACH program. Stakeholder meetings revealed that Ministry of Health, Gender and Community Development, Agriculture and other government departments worked closely with sub-grantees to complement and/or fill existing financial and material resource gaps. The government provided HIV test kits, drugs, IEC materials, technical guidance and other forms of support to sub-grantees such as facilitation of training to volunteers and peer educators, sharing of transport costs, providing leadership and many more. Thirdly, the activities sub- granting supported were culturally sensitive. For example, HIV Prevention Strategy on the use of condoms did not target children under the age of 12. Issues of stigma and discrimination were also culturally handled among other vulnerable children. Door to door approach also proved to be ‗culturally‘ appropriate in a society where issues of sex are still highly perceived as private. The door-to-door strategy, therefore, provided privacy and an appropriate environment for people to open up and ask questions. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 54 Fourthly, the nature of activities sub-granting supported addressed specific problems of individuals and communities at large. In all focus group discussions conducted, there was a sense of ownership of Community REACH funded program activities because of evident improvements in the quality of life, health, self-esteem and dignity in the lives of OVCs and People Living with HIV/AIDS. Participants in focus group discussions testified that the activities were needs-centred and reached those who most needed them. Some programs such as HTC, CPMTCT created more demand to an extent that supply of goods (condoms and HIV testing kits) and services (volunteer follow ups) became increasingly limited. Fifthly, the program objectives were achievable within the given time frame. The nature and impact of activities contributed to the objectives of Pact Malawi‘s program and strategic objectives of USAID. 6.3 Financial Management Support to Sub-partners Over the 5 year period of its program, Pact Malawi had 26 sub-grantees solicited and unsolicited from across the country. Two of the sub-grantees were suspended due to non-compliance and financial mismanagement. The success of administration of sub-granting depended a lot on Pact Malawi‘s staff availability, expertise and experience. Pact Malawi ensured adequate staffing to administer the grants and took an initiative to hire five new members of staff critical for this program comprising Senior Program Manager, MER officer, Quality manager, Program Officer and Finance and Grant Assistant. Pact Community REACH Program had two Finance officers and two Monitoring and Evaluation officers. Among other forms of support, Pact Malawi conducted financial management and system training, administrative close-outs, compliance reviews. This assisted sub-grantees to improve on financial management systems. As of September, 2011, financial management systems (accounting procedures, internal controls, and budgeting, reporting and auditing and policy environment) of sub-grantees had increasingly improved compared to Pre-award assessment period (Pact Evaluation, May 2011 pp 11- 12). A review of technical documents indicated that Pact Malawi‘s financial team was responsive and supportive to the technical needs of sub-grantees in aspects of financial management. Short falls in budgeting, reporting and auditing could be attributed to irregular feedback and follow ups on finance officers as this was a common complaint among most financial officers. Partners also complained of monthly reporting system which to them affected program implementation because of the long time involved in compiling reports. As a result of the vigilance of the Pact Malawi program and grants management team, the program was able to identify non-compliant partners and this led to cases of fraud detected. To its credit, Pact Malawi‘s further review of the situation resulted in 2 organisations (Tutulane and Zomba Catholic Health Commission being suspended and in one of these cases a member of staff was found guilty and imprisoned for three years. 6.4 Pact’s Partner’s Rating of Sub-Granting All partners spoke highly of the sub-granting in how it assisted organizations and beneficiaries increasing their capacities and meeting specific needs of beneficiaries. While sub-granting received praises from nearly all partners, it was criticized for being inflexible and inefficient in certain aspects. Sub-granting had a fixed exchange rate as a result it did not respond to changing economic environment and needs of beneficiaries. For example, cost of fuel, rising inflation among others, limited sub-grantees‘ ability to implement some of the much needed activities. Inefficiencies, on the other hand, resulted from delays in transferring of funds to sub-grantees. This was a major concern among all sub-grantees because it impacted on their programming. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 55 6.5 Lessons learnt as a grant recipient  Effective grant management: transfer of funds based on projected expenses and on the backdrop of expenditures in the previous period  A combination of resources and skills result in improved performance  Timely submission of report is important  Discipline in financial management  Partnership promotes service delivery  Community empowerment promotes sustainability  Regular coaching and mentorship for improved accountability  The importance of month to month tracking  Emphasis on procedures helped checking and controlling expenditures  Monthly disbursements affects program activities  Dependency on donor funding not appropriate  Importance of compliance to project obligations  Training helps in prudent management of grants Table 4 below shows efforts which both Pact Malawi and sub-grantees put in place to ensure financial prudence. Table 4: Pact Malawi system strengthening and Sub-grantee system strengthening Pact Malawi’s Systems strengthening Sub-Grantees system strengthening  Pact Malawi made periodic supervision, mentoring visits, audits and was making sure that disbursement of next funds was upon satisfaction of the previous expenditure on monthly basis  Conducted vouching on quarterly basis  introduced a procurement section backed by financial management policy  Pact Malawi established internal controls: five authorized signatories that sign cheque when paying for materials or services  Pact Malawi involved external auditors who came to audit financial records  Submission of Monthly and quarterly financial reports.  Quarterly Physical checking of finance files that contained budgets, biding documents, local purchase orders, invoices, requests, payment vouchers, receipts and other relevant information  Stamping of any financial document checked by Pact Malawi  Pact Malawi provided supportive visits to provide technical assistance  Periodic data auditing  Financial policies and systems  Briefing staff on the funds received and sharing budgets with responsible officers on different activities  Development of an operational/work plan  Requisition of funds by program coordinators and requisition being reviewed and checked by program manager, certified by FAM and then approved by Executive director  Writing of voucher by accounts assistants  Voucher is signed or certified by FAM and the approved by the ED  Cheque production (all payments are done through cheques.  Liquidation of funds by coordinators  Updating the cash book  Getting bank statements and producing monthly financial reports  Involvement of external auditors like Graham Carr was also a milestone. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 56 SECTION SEVEN 7. PACT MALAWI COMMUNITY REACH PROGRAM GENDER INTEGRATION AND CONSIDERATIONS Pact Malawi advocated for gender mainstreaming and incorporation into its program. This was reflected in the Annual Program Statements which Pact Malawi issued. They laid bare their intent for gender considerations in all the proposals partners submitted to them. Pact recognised that while most civil society organizations providing services to populations affected and infected by HIV/AIDS had heard about the importance of gender mainstreaming from the government, donors and peers, they were not fully prepared to mainstream gender in their HIV/AIDS programs and in their organizations. Most program officers and managers equated gender mainstreaming with ensuring participation of equal numbers of men and women in various activities. In short, gender mainstreaming was seen as a counting exercise. Furthermore, there seemed to be a lack of understanding of the importance of promoting gender equality as well as equity. Pact‘s approach and efforts effectively and efficiently enhanced the understanding, by its partner organizations, that gender mainstreaming involves more than having equal numbers of men and women in current structures and activities; but that it involves changing institutions, policies, programs, projects and processes so that they promote gender equality. Gender inequalities in Malawi have been identified as one of the factors increasing vulnerability to HIV for women. The HIV prevalence rate is higher among women (13.3%) than men (10.2%). Several factors have been identified as putting women at risk, one of them being power imbalances between men and women leading to women‘s inability to negotiate condom use. With this knowledge and in order to make a change in this area, Pact Malawi organized a Gender Mainstreaming in HIV/AIDS programming training for its REACH partners in order to enable partners to integrate gender issues in their programming and organizations. Factors accounting for women‘s vulnerability to HIV/AIDS were explored during the gender mainstreaming training using various adult education methodologies such as role play, group discussions and small group work. The training aimed to be transformative in that it not only looked at societal and organizational challenges around gender but also helped participants to explore the role of gender in their lives (at their home, workplace, city or village). Pact Malawi‘s gender focus, principles and training enabled sub-partner staff to have a clear understanding and application of gender mainstreaming concepts and strategies in their programs and organizations. Community REACH program also provided local organizations with the tools they could use to mainstream gender in their programs and organizations. Pact Malawi‘s program approach on gender helped the local organisations to systematically ensure that gender considerations were placed at the centre of their policies, programs, processes and activities. Sub-partners‘ gender analysis knowledge and skills were strengthened and they were able to integrate gender concerns in the analysis, formulation and monitoring of policies, programmes and projects, and the systems that supported them, in order to promote gender equality and the empowerment of women in their program activities. To that effect the program registered some positive gender outcomes. The program saw the emergency of male champions especially at MAICC in Dowa whose mandate was to strengthen the referral systems with other HIV-related services such as Prevention of Mother to Child Transmission (PMTCT) services. This initiative was conceptualized in order to combat gender imbalance in using and utilizing HIV services which prior to the formation of Male champions clubs were dominated by males. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 57 The Male Championship clubs did not only fight the gender imbalance but also advocated for couple counselling and testing which made tremendous in-roads as more couples were going for counselling and testing and depending on the outcomes of their tests they sought appropriate HIV and AIDS services as a couple. The male championship club proved to be a success and more males were willing to join than when it was just introduced as remarked by one of the participants during the focus group discussion. ―When the club was just introduced here at Dzoole, people were laughing at us and labelled us with all sorts of derogatory names because they thought we departed from our norms by advocating for and doing things which were predominantly viewed as female orientated. We never relented with such remarks and over the years when the community and those who were laughing at us saw the gains our families made in accessing HIV services and improved quality of life, the attitudes and perception of people completely changed. We all suddenly became champions and many people were attracted to what we do. To date we continue registering new members and the demand to become champions is overwhelming.‖ —Male Champion club member The gender consideration has also seen breaking some cultures that increase the risk of women to contract HIV. For example, at NACC through gender considerations of the program helped to reduce early marriages for girl children, a culture which was deeply rooted amongst the people of the area. During the role out of national OVC quality improvement standards at NACC, the project in collaboration with local leaders established by-laws to encourage parents to send their children to school and also work on those that were involved in forced marriages to go back to school. Following this development 11 girls abandoned marriages and went back to school and the practice of forcing girls into early marriages is gradually and continually being challenged by the community members themselves. Other harmful practices were discouraged including wife inheritance in Mzimba district; stigma and discrimination and how this affected issues of disclosure among couples as noted in Nkhotakota north under NASO. Mentoring and coaching efforts of Pact Malawi program staff helped sub-partners understand the importance of putting emphasis on interrelationships between theories and ideas about development and the organisations and people making key decisions. Pact Malawi partners developed high recognition that the ideas and practices in the mainstream determined who got what and provided a rationale for the allocation of community resources and opportunities. Gender Mainstreaming Success Story Nellie: The woman who wants to save the world In the Nkwalula village within the area of Namwera in the Southern region of Malawi, 2005 wasn‘t a very good year for most of its inhabitants. Children, women and men were dying and they all seemed to die from the same disease. Someone had to do something. Nellie Kandulu decided to be that person. During the World Aids Day official event of that year, Nellie decided to tell her story, hoping to save her fellow villagers. She stood up from the crowd and told everyone that she was living with AIDS. ―A lot of people were dying because they were afraid of asking for help. That is self stigma and denial. I realized that I had to come up and tell my story so that I could also get the skills on how to go about it, and how to help others,‖ explains Nellie. And it helped. After her confession, a lot of people started flocking her home because they knew her, saw her when she was healthy and saw her when she was sick as well. They asked her a lot of HIV-related questions, and ever since, she has helped to establish 14 support groups for people living with HIV. Nellie, who is currently the chairperson for the Titandizane support group (which means let us help each other in the local language), is now at the forefront of encouraging her fellow women to be self reliant, saying: ―husbands could simply disappoint you one day.‖ Behind those words, there is a sad story of her disappointment. ―I was married in 1981 and separated from my husband in 2000. My husband is currently Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 58 living in South Africa and he has remarried. However, I don’t get any assistance from him and have to take care of the six children we have together, all by myself,‖ she says. What increased her suffering was the situation she was in when she was diagnosed with HIV in 2004 after being sick on and off for quite some time. ―I was all alone when I received the results, and it made me feel even more lonely and sad,‖ she explains. She was encouraged by her doctor not to be upset and instead take the drugs according to the instructions. Namwera AIDS Coordinating Committee (NACC) encouraged Nellie to form a support group, and she started with five other fellow women after informing her village leaders and asking them to direct other people living with HIV to join their group. NACC has also supported Nellie in the House Economic Strengthening Program by providing her with goats. Through the cattle, she is now able to sell those that are being reproduced and can pay for her children‘s school fees. She also received a small loan to start a business of baking scones. Says Nellie: ―I can now assure everybody that being HIV positive doesn’t mean that it’s the end of life. I was sick and bony, but look at me right now…I’m independent and self reliant. It’s not the end of the world. You can do something!‖ The courageous lady is now an HIV volunteer who also goes to other areas to encourage others through NACC, and advocates for the establishment of HIV Testing and Counseling centres in those areas. When she counsels, she never hides the fact that she is living with HIV. ―Some people normally ask me why I am so courageous, and if I‘m not afraid of spoiling my CV. But I tell them that my aim is to help people. I feel it is my duty, my responsibility to help my fellow friends,‖ says Nellie, adding that she loves her work as a volunteer and doesn‘t need anything else in return other than committing to rescuing people till the end of her life. Praise for NACC never leaves her lips: ―Today NACC and I are like a finger and a ring. They have done a lot for me: they motivated me, changed me, and have helped make me fat. People now say that I am not HIV positive, and that I‘m just lying as part of a campaign,‖ she explains, laughing. And sure enough, Nellie Kandulu is a household name in Nkwalula village. She is strong, courageous, and wants to help people come forward and stop the self-denial. Yes, she wants to save the world. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 59 SECTION EIGHT 8.0 SUSTAINABILITY Pact Malawi‘s approach and strategies ensured and enhanced the sustainability of its local partners and their program activities in several ways. Pact contributed to its partners towards the move to financial self-sufficiency and sustainability by encouraging them to incorporate income generation activities into their activity plans. Pact Malawi worked with each of its partners to establish Data Quality Teams whose membership included officials at district level of government and civil society agencies. This enhanced participatory monitoring at partner level. Sub-partners were supported and encouraged to form collaborating and networking alliances with various stakeholders at district and community levels including District Aids Committees, District Health Offices, District Health Management Teams, Village Aids Support Groups and Committees. These alliances will enhance the supervision of VCT sites, supply of testing kits and condoms. In some cases, resources such as offices and equipment are shared contributing to the cost efficiency of activities. The Community REACH program in Malawi gave evidence of the importance of the USAID‘s and PEPFAR‘s overall vision for local organizations globally to continue building capacities of local organizations and ensure sustainability of development programs through increased local ownership of programs. This was achieved because the local organizations (sub-partners) in Malawi were strengthened and that their capability of managing more and more resources as were as delivering strong technical programs was enhanced. Pact Malawi Community REACH program helped USAID Mission to provide a platform to grow and build capacities of local organizations. Pact Malawi provided fundraising strategy training and proposal writing training and information on potential financing agencies was shared with Pact Malawi partners along with the grant writing training. In-depth assessments revealed that while most sub-partners were actively working on mobilizing resources, several had not secured additional resources to fully cover for the activities that were funded by REACH. During the time of evaluation, most sub-partners had applied and had been awarded funds by National Aids Commission (NAC) in 2010, but funds had not yet been disbursed. Some funders are pulling out of Malawi or reducing development assistance. This is affecting direct funding and technical support to civil society as well as the support civil society receives from government since the government funds that come from direct assistance from donors are draining and government in turn has fewer resources to support civil society. To their credit, as a result of the capacity growth, some of the sub-partners developed the ability to mobilize resources directly from USAID and other bilateral development partners. As a result, the sub-partners that implemented this program have the potential of becoming bigger, better and well able to deliver high quality services. This development and evidence of growth will contribute to the sustainability of the local organizations and their work in Malawian rural and peri-urban communities. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 60 SECTION NINE 9.0 CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS 1. The findings of the evaluation indicate that Pact Malawi Community REACH program achieved its overall objective which sought to empower local organizations to support HIV and AIDS programs that reach individuals, families, and communities most vulnerable to HIV infection and HIV-related consequences with the services they need most. 2. Pact Malawi‘s Capacity building through mentorship was an effective way of transferring knowledge and skills to partners and was an excellent supplement to formal training. 3. Pact Malawi‘s Community REACH program largely attained its targets for persons that received capacity development and technical assistance and also people that received the services, including scaling up sites providing PMTCT, HTC, HBC and Pre-ART services. 4. Pact Malawi‘s support to its partners demonstrated that a local NGO with sound and transparent financial management will build confidence in the community it is serving as well as attracting local and international donors to fund it. 5. Networking and collaboration among Pact Malawi partner organizations is a key for success. Networking provides a sharing of strategies and best practices and allows service providers to acquire and enhance their service delivery skills, use appropriate and relevant techniques and avoid duplication of interventions. 6. Pact Malawi achieved excellent working relations with the MOH and other stakeholders at national, regional and local levels. While these relations enabled Pact Malawi to make significant contributions to achieving its targets, they also fostered, to a less degree, feelings of dependency by few sub-partners. Factors included limited financial support sources (USAID was the main finance contributor for most of the partners); medical supplies were mostly from the ministry of health that was also facing serious shortages. The follow-on activity must address this dependency from the onset and articulate a clear exit strategy in the design. 7. The Community REACH program has had substantial success in rapidly enhancing the response to HIV/AIDS in Malawi. Pact‘s activities provided a solid base in terms of human resources and infrastructure in different local organizations and in the community from which to expand USAID Malawi‘s care and support initiatives. 8. Pact Malawi‘s establishment of a monitoring and evaluation department enabled organizations to keep better track of program data. It also allowed for this kind of data to be more easily accessed and useful. It is essential that monitoring data be used to inform program decision making. 9. The absence of baseline data for HIV and AIDS outcome indicators made it difficult to measure progress against benchmark indicators. As a result progress was measured against estimated targets by Pact Malawi staff and their partner agencies‘ counterparts. 10. The capacity building efforts of partner organisations by Pact Malawi illustrated that the development of a strategic plan creates organizational focus and draws staff to understand activities that are planned and reasons they are necessary. It keeps everyone on track and can be used as a tool to mobilize resources. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 61 11. Pact Malawi Community REACH program approach with its partners demonstrated that capacity building of local communities is essential for project sustainability and ownership and increased service uptake. 12. Pact Malawi capacity development support that combined both technical as well as organizational development proved effective in improving local organisations. OD interventions supported partners to improve their organizational systems, leadership and management, strategic planning, governance and monitoring and evaluation among others. These were areas of great need at the beginning of the program. The technical aspects improved their service delivery skills. By combining the two, the interventions became richer and more effective. 13. Formation of data quality teams by Pact Malawi partners which constituted cross functional teams made up of staff, local leaders and other stakeholders with different levels of responsibility made the results and achievements of partners rich because this promoted shared ownership and responsibility for ensuring that the compilation and reporting of results was accurate and well documented. RECOMMENDATIONS 1. Partners whose capacities have reached a level of maturity with the support of Pact Malawi (these include organizations such as NASO, Malamulo, NACC, COPRED, CCC, MAICC, Ekwendeni Hospital and Tovwirane) should be supported to get direct funding from USAID. These agencies have teams of qualified personnel with vast experience working with many donors. They have functional boards of trustees and sound financial and administrative systems. 2. In future, PEP should be modelled around the community PMTCT concept which has proved to be effective and with a functional follow –up procedures and where defaulters are quickly identified and brought back to the treatment. 3. For the next generation of indicators of care, there is need to split the beneficiaries of OVC and PC/HBC. This will necessitate understanding of how many people have benefitted from OVC or PC/HBC. 4. In the next activity, continue to provide HTC outreach where at-risk youth congregate, or where other most at-risk persons live or work, to increase uptake by those most in need of care, support and prevention interventions; In addition, peer to peer HIV prevention education should be scaled up. 5. Future USAID-funded projects should seek to include Health systems strengthening component that deals with supply chain management to make sure that supplies like test kits, condoms and other drugs are available at all times to beneficiaries. It was observed that Pact Program successfully created demand for services but the efforts were undermined by intermittent drug supplies. The health systems strengthening should also consider strengthening the human resources for health. 6. Future program design should include baseline assessments for HIV and AIDS outcome indicators to establish benchmarks in order to track progress against set objectives, outcome and impact indicators. 7. Capacity development initiatives require resources in terms of time, material, human and financial. All capacity development initiatives in the future should be attached to budget to avoid the situation where by partners plan for the initiatives but never implement them because of lack of resources. This limitation should be taken into account in the design of future programs of similar nature to the Pact Malawi Community REACH. Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 62 ANNEX 1: Reference documents 1. "A cross-sectional evaluation of the HIV prevalence and HIV-related risk factors of men who have sex with men (MSM) in Malawi", E. Umar, et al. AIDS 2008 - XVII International AIDS Conference: Abstract no. MOPE0412 2. Agreed upon procedures review of USAID resources managed by Pact Malawi Community REACH program (Period October 1, 2007 to June 30, 2009: 21 June 2010 3. Beyrer, C et. al (2010, August) 'Bisexual concurrency, bisexual partnerships, and HIV among Southern African men who have sex with men' Sex Transm Infect 86(4) 4. G A Bello, J Chipeta and J Aberle-Grasse (2006), 'Assessment of trends in biological and behavioral surveillance data: is there any evidence of declining HIV prevalence or incidence in Malawi?' 5. Malawi HIV/AIDS extended National Action Framework (2010 – 2012) 6. Malawi HIV/AIDS National Action Framework (2005- 2009) 7. Malawi National Guidelines and PEPFAR guidelines for key HIV/AIDS technical areas 8. Malawi Partnership Framework between the Government of the United States of America and the Government of the Republic of Malawi (2009) 9. Malawi/PEPFAR Partnership Framework and Implementation Plan 10. Pact Malawi Capacity Building Initiative Evaluation Report May 2010 11. Pact Malawi Community REACH Program Description and Cooperative Agreement 12. Pact Malawi Organizational Capacity Re-Assessment: summary Report July 2011 13. Pact Malawi Performance Monitoring Plan Revised 2009 14. Pact Semi-annual and Annual reports 2007 to 2011 15. PEPFAR Guidance Documents 2007 - 2011 16. PEPFAR Next Generation Indicators Reference Guide Version 1.0 June 2009 17. PMTCT, HTC, HBC, OVC, HIV sexual prevention 18. UNAIDS (2008) 'Report on the global AIDS epidemic' 19. UNAIDS (2008) 'Report on the global AIDS epidemic' 20. 'UNAIDS (2010) 'UNAIDS report on the global AIDS epidemic' 21. UNGASS Country Progress Report (2010) 'Malawi HIV and AIDS Monitoring and Evaluation Report: 2008-2009' 22. UNGASS Country Progress Report (2010) 'Malawi HIV and AIDS Monitoring and Evaluation Report: 2008-2009' 23. United States Government Malawi Emergency AIDS program Strategic Plan (2006) 24. USAID Health Team Strategic plan 25. WHO/UNAIDS/UNICEF (2011) ‗Global HIV/AIDS Response: Epidemic update and health sector progress towards Universal Access 2011 Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 63 ANNEX 2: Evaluation Schedule Date Activity Location Organisation 5 October Finalise tools Lilongwe Salephera 6 October Planning & strategising Lilongwe Salephera 7 October Team meeting Lilongwe Salephera 18 – 19 October Pretesting Nkhotakota SWAM 20-21 October Organisational review FGD Nkhotakota NASO TEAM South: Esnart Bello & Dr. Bagrey Ngwira Date Activity Location Organisation 24-25 Organisational review, FGD Lilongwe Nkhoma CCAP hospital 26-27 PACT Partners Review Meeting Lilongwe Pact & USAID 28 Organisational review, FGD Lilongwe Partners In Hope 31 Oct – 1 Nov Hestern & Alfred Organisational review, FGD Lilongwe Lighthouse 31 Oct – 1 Nov Organisational review, FGD Thyolo Malamulo 2-3 Nov Organisational review, FGD Thyolo Christian Community Church (CCC) 4-5 Nov Organisational review, FGD Blantyre COPRED 7-8 Nov Organisational review, FGD Blantyre AHS 9-10 Nov Organisational review, FGD Mangochi NACC 11 Nov Travel to Lilongwe TEAM NORTH: Sosten Chilumpha & Nina Chihana Date Activity Location Organisation 24-25 Oct Organisational review, FGD Dowa MAICC 26-27 Oct PACT Partners Review Meeting Lilongwe 28 Oct Organisational review, FGD Lilongwe NAPHAM 31 Oct – 1 Nov Organisational review, FGD Lilongwe Nurses & Midwives Council of Malawi 2-3 Nov Organisational review, FGD Mzimba Tovwirane 4-5 Nov Organisational review, FGD Mzimba / Mzuzu Ekwendeni Hospital 7-8 Nov Organisational review, FGD Rumphi FVM Matunkha 9 -10 Nov Organisational review, FGD Karonga Lusubilo 11-18 November Analysis and report writing Lilongwe 19 November Submit draft report Lilongwe 25 November Submit final report Lilongwe List of National Stakeholders consulted Name Organization 1 Edith Nkawa Global Fund coordinating Secretariat 2 Willard Manjolo Ministry of Gender 3 Elias Michael Ministry of Health 4 Glory Mkandawire Bridge Project 5 Mr. Teleka National Aids Commission Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 64 ANNEX 3: Pact Evaluation Focus Group Discussion Guide With Communities Name of Community: ---------------------------------------------------------------------------------- Name of PACT Sub-Partner: ------------------------------------------------------------------------- Signature of Community Representative:---------------------------------------------------------- Date: ------------------------------------ Time:------------------------------------------------ Name of Facilitator: Name of Moderator: Date of FGD: Start Time End Time Prevention-Abstinence and Being faithful For beneficiaries of AHS,CCC, COPRED, Ekwendeni, Lusubilo, MAICC, FVM – Matunkha, NAPHAM, NASO, Nkhoma Synod, Partners in Hope, Tovwirane, Malamulo, NACC (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. Briefly describe HIV/AIDS prevention initiatives in your community in respect to youths and adults? 4. What HIV/AIDS prevention messages have helped change behaviour of the youth and adults? If no, why 5. How were the HIV/AIDS prevention messages packaged and conveyed? 6. How did the prevention messages help change behaviour of the youth and adults? 7. How do you currently understand and practice preventive behaviours pertaining to abstinence and being faithful? 8. What in your view has been the most effective approach in disseminating HIV/AIDS prevention messages? why 9. Explain why other approaches have not been effective? 10. What do you think should happen to make the approaches effective? 11. What comments do you have on the design and content of IEC materials? Other Prevention For beneficiaries of COPRED, Ekwendeni, MAICC, FVM – Matunkha, NAPHAM, NASO and PIH, Tovwirane, Malamulo, NACC (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. a. Describe how you view the work of condom distribution, use and sensitization in this community in reaching the most at risk groups? b. Do people use condoms here? 4. In your opinion, would you say there has been adequate counselling on condom use? 5. How receptive has the community been on issues of condom distribution and sensitization? 6. How has the community‘s response (involvement) helped advance the work of condom distribution and sensitization? 7. What in your opinion have been the successes and challenges of condom distribution and sensitization? 8. What are the barriers to condom use in your community? 9. How would you describe the role of the youth and most at risk groups in this initiative? 10. What positive and negative impacts have you noted on the youth and most at risk population as a result of this initiative? 11. Which has been the most difficult group to reach and why? 12. What are the barriers in reaching out to the target populations? 13. What best strategy would you suggest to ensure this group/target is reached? Most at Risk Population (MARP) For beneficiaries of NACC, Tovwirane, Malamulo, Matunkha (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 65 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. Who do you consider to be the most at risk people to be infected with HIV in your community? 4. Are they reached out with information about HIV prevention? 5. If yes, who reaches out to them with the information? 6. How do you think these prevention messages has helped the most at risk groups? 7. Is there any behaviour change among these ―most at risk‖ people? PMTCT For beneficiaries of Tovwirane, NASO, MAICC, Nkhoma, Malamulo, NAC and AHS (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. Briefly describe PMTCT services in this community in terms of access and service delivery? 4. In your experience, what has been the most hindering factor in accessing PMTCT services? 5. What in your view is lacking in delivery of services? 6. What has made PMTCT a success in your community? 7. How would this problem be solved? 8. How would you describe the work of community mobilization in respect to efforts to increase demand for PMTCT services? 9. To what extent is the male involvement in the following: a) Ant Natal Services b) Access and support for ARV prophylaxis c) Post natal services 10. What do you think has been the positive impact of male involvement in PMTCT services? 11. Are there women in this community who received these special services but discontinued? If yes, why? 12. Are there any efforts by volunteers or program staff to ensure those discontinued services were reintegrated into service provision? What was the outcome of the efforts? For Partners doing community PMTCT 13. When did you start implementing the community PMTCT? 14. What are some of the benefits the community have registered in this program? 15. How effective is the referral from community to the health facility? Post Exposure Prophylaxis (for Nurses and Midwives Council of Malawi). This questionnaire will be administered to participants who attended PEP training and with the help of PACT, selected participants will be gathered for a focus group discussion. 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. How was the training delivered? 4. To what extent do you get exposed to HIV infection during work? 5. In case of exposure, what medical and counselling services do you receive? 6. How do you assist other clients e.g. rape cases? 7. How quick do you access the services? 8. How helpful have these services been? 9. Is there anything that needs to be done to make the services more effective? 10. What are some of the lessons learnt? HTC for beneficiaries of AHS, Ekwendeni, MAICC, Malamulo, FVM – Matunkha and NACC, Nkhoma, NASO, Tovwirane, MAICC, COPRED. (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. How have community mobilization campaigns about HTC been helpful to your community? 4. Where do you access HTC services in this community? 5. Briefly explain the quality of service delivery of HTC in the mentioned outlets above?(Staff, equipment, availability of test kits, Confidentiality etc) 6. What are some of the challenges you experience at HTC centres? 7. What are the community‘s attitudes towards VCT? 8. How would you describe level of community participation in HTC mobilisation campaigns? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 66 9. What are some of the factors that reduce increase/reduce community participation in this initiative? 10. What are some of the lessons learnt? Palliative Care/ Home Based Care For beneficiaries of COPRED, Ekwendeni, Light House, Lusubilo, MAICC, NAPHAM, NASO, Tovwirane, Malamulo, NACC, Matunkha (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. What activities does this organization do in your community in relation to Palliative care/HBC? 4. To what extent have the support groups and post-test clubs assist in improving quality of life of PLWHAS? 5. On a scale of 1 to 10, how in your view do you rate the services of HBC care givers in your community? ( on issues of nutrition, administration and access of ARVs, management of opportunistic infections) 6. How receptive are the PLWHAS to these support services? 7. On a scale of 1-10, how would you rate the organisation‘s commitment in their work with PLWHAs in your community? 8. Are there any referrals or networks between home, community services and health facilities? 9. On a scale of 1 to 10, how would you rate these referral services? 10. What positive changes have taken place in the lives of PLWHAS after accessing these services? Orphans and Vulnerable Children For beneficiaries of COPRED, Lusubilo, MAICC, NACC, NAPHAM, NASO, Tovwirane, Matunkha (Beneficiaries include: Direct community members, community beneficiaries, youth groups, drama groups agogo groups, male champions) 1. Are you aware of (name of organization)? Have you worked with this organization? Have you ever received support from this organization? What kind of support did you receive? 2. How did the support you received help you? 3. What activities does this organization do in your community in relation to OVCs? 4. On a scale of 1 to 10, how do you rate the quality of OVC services by this organization? 5. What positive changes have taken place in the lives of the OVC as a result of these activities? 6. In your opinion, how have the activities addressed different needs of the OVCs in this community? (issues of physical, emotional needs etc) 7. In your opinion, how have the initiatives helped to change the trauma/shock level and deviant behaviour in the OVCs? 8. What type of life skills have been imparted in the OVCs and to what extent have these skills improved their lives? 9. What support do the community/families give to OVCs to ensure continuity of OVC initiatives? 10. What in your opinion is the level of participation in life skills training among OVCs? 11. What are some of the challenges faced by OVCs not to adequately access like skills training? 12. What do you think should be done to enhance OVCs a participation in life skills training? 13. What are some of the lessons learnt? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 67 ANNEX 4: Pact Program Evaluation Capacity Development Questionnaire for Sub-partners This questionnaire is addressed to Pact sub partners which are local NGOs that Pact was supporting including Global fund Coordination Mechanism secretariat. Purpose: To assess the nature, quality and impact of capacity development assistance (OD support) provided to recipient organizations by Pact under the Community REACH programme. Respondents: Respondents of this questionnaire are board members, Executive Directors, programme managers, program officer, project/finance officers. Name of Organization: Names and positions of person (s) being interviewed: What role did the person being interviewed play in the Community REACH programme? Name of interviewer: Date of interview: Capacity building initiatives 1. What type of capacity development initiatives did Pact implement in your organization since 2007? (Below are some of the capacity development initiatives that Pact implemented) probe if the partner does not bring out an exhaustive list. Organization capacity assessments (OCA) Multi partner capacity building events( Targeted interventions RE-OCAS Peer learning Institutional strengthening plans Mentoring Coaching Trainings Provision of reporting templates Provision of feedback on quarterly reports Data Quality Assessment (DQA) Organization Network Assessment (ONA) Support visits Review meetings Other Health Systems Strengthening (OHSS) 2. How was it decided that you needed assistance in that area? 3. Would you please give us a brief explanation of each intervention, which areas did it address? On a scale of 1 to 10 how helpful and effective was each intervention to your organization? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 68 Type of intervention What it is Target areas How effective it was( rate the effectiveness on scale of 1 to 10 1 Organization capacity assessments (OCA) 2 Multi partner capacity building events 3 Targeted interventions 4 RE-OCAS 5 Peer learning 6 Institutional strengthening plans 7 Mentoring 8 Coaching 9 Provision of reporting templates 10 Training 11 Provision of feedback on quarterly reports 12 Data Quality Training 13 Review meetings 14 ONA(Organization Network Analysis) 15 Support visits 16 Other Health Systems Strengthening (OHSS) 4. To what extent has each initiative brought change in your organization and what are the changes? 5. How were the results of each of the initiatives provided assessed? Were there regular feedback and follow-up sessions? 6. If you have to rate the capacity development initiatives, which was the most effective and which one was the least effective and why? Type of intervention What change did it bring to your organization What was the most effective to your organization (from 1 to 16) 1 Organization capacity assessments (OCA) 2 Multi partner capacity building events 3 Targeted interventions 4 RE-OCAS 5 Peer learning 6 Institutional strengthening plans 7 Mentoring Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 69 8 Coaching 9 Provision of reporting templates 10 Trainings 11 Provision of feedback on quarterly reports 12 Data Quality management trainings 13 Review meetings 14 ONA(Organization Network Analysis) 15 Support visits 16 Organization Health Systems Strengthening (OHSS) 7. Pact concentrated on ten areas of organizational development as one of their capacity development initiative to your organization. And from document review, we have learnt that your organization has improved most and least in the following areas. (Specifically mention the areas the partner has improved most or least. The areas are strategic direction, governance, material and finance resources, organizational culture, Human resource management, knowledge management, external partnerships, administration, program management and service delivery, advocacy and lobbying.) Why has your organization improved most or least in these areas? 8. For the areas that you improved least, what were the challenges? What were some of the challenges for the Capacity Development Program (CDP) as a whole? 9. What are some of the shortfalls of the capacity development programme as a whole? 10. Pact‘s support was the combination of sub-granting, technical and capacity development support to local NGOs. On a scale of 1 to 10 how would you rate the effectiveness of this programme as a whole? What would you suggest differently to make it more effective? Do you feel there are some areas that the programme did not address in terms of capacity building? What are the areas? 11. On a scale of 1 to 10, how do you rate the effectiveness of the capacity development support that you received from Pact? What were the challenges and gaps? How were they addressed? Capacity Development service providers 12. What about the Capacity Development service providers, CABUNGO and pact staff, what are the strengths of each of them in their delivery of the services? CABUNGO Strengths Weaknesses Pact Staff Strengths Weaknesses 13. Are there any areas that need improvement in their delivery of the services? What are they? Collaboration and linkages 14. In terms of collaboration and linkages, who do you collaborate and link with and on what areas? How useful are the linkages and collaboration? Partner Service area Area of collaboration Sustainability and service delivery 15. How will the capacity development initiatives that you received from Pact help you in the future since Pact‘s funding has expired? 16. How has this capacity development assistance improved implementation of your activities? Please explain 17. Apart from Pact Malawi, did you manage to get other funding from anywhere else? Who are the donors and how long have they funded you and how much? 18. How has your organization planned to sustain the activities after Pact? Any exit strategies in place? What are they? Lessons learnt 19. What are some of the lessons learnt as a recipient of capacity development support provided to your organization by Pact? Sub-granting support 20. How much funding did you receive from Pact? 21. Out of the money that you received, how much went administration and how much went into activities/programmes? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 70 22. What process did the organization go through to be enlisted as one of the recipients of Pact‘s funding? 23. What processes did you go through to receive funding? How different were they from other donors? 24. How fast did you receive funds after sending a requisition for funds or after meeting the requirements? 25. What mechanism did you put in place to ensure that money was put to use prudently? 26. What mechanism did Pact put in place to ensure that your organization is accountable in the use of funds? 27. On a scale of 1 to 10, how do you rate the effectiveness of the sub-granting support that you received from Pact? What were the challenges and gaps? 28. What are some of the lessons learnt as a grant recipient of Pact? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 71 ANNEX 5: Pact Program Evaluation Questionnaire for Pact Management and Staff This questionnaire is to be administered to individual members of Pact Program Management and Staff. (Country Director, Monitoring and Evaluation manager, finance manager, program officers, Monitoring and evaluation officer, finance officers) A. PROGRAMME DESIGN ISSUES From management and staff perspectives: 1. Which elements of the program worked well and why? 2. a. What necessitated the need or reason for program modifications? How effective were these modifications? (b. How did you implement the condom dispenser program?) c. What were the challenges? 3. What other alternative ways or approaches of achieving the project objectives were available, other than the one (s) used? Briefly explain. B. EXTENT TO WHICH PLANNED INPUTS & RESULTS HAVE BEEN ACHIEVED 4. How did Pact determine types of TAs to a particular sub partner? 5. What criteria did you use to select sub partners? 6. To what extent has the project achieved its objective related to HIV/AIS technical support? 7. How was the technical support provided? 8. How effective and efficient was the program in terms of organization structures in administering HIV/Aids technical support? 9. How effective and efficient was the program in terms of organization structure in administering local grants? 10. How effective and efficient was the program in terms of organization structures in administering capacity building to local NGOs? 11. How did the project activities reach a high proportion of beneficiaries of various sub groups such as women, men, and young people, PLHIV and OVCs? 12. To what extent was gender integrated in the program? And to what extent was it effective? A. EFFECTIVENESS AND EFFICIENCY [Effectiveness answers the question, ―Is it good value for money?‖ are total programme costs worth the programme results while Efficiency answers the question ―could these programme results been achieved at a lower cost?‖] 13. On a scale of 1 to 10 how would you rate PACT‘s effectiveness and efficiency of the sub-granting support to NGOs? In relation to this, what were the challenges and gaps? a. How much funding was allocated to the whole program and how much funding in total was disbursed to sub partners? b. Out of the money that went to sub partners, how much money went into program work and how much went into administration? c. What processes and procedures did the sub partners need to fulfil for them to get a disbursement? How difficult or easy was it for the sub partners to receive funds from you? Why? d. What control measures did Pact put in place to ensure accountability in the use of funds e. What were the challenges and gaps in the sub-granting support to sub partners? 14. To what extent would you rate Pact‘s effectiveness and efficiency of the capacity building support to NGOs? In relation to this, what were the challenges and gaps? 15. To what extent would you rate Pact‘s effectiveness and efficiency of the technical support to NGOs? In relation to this, what were the challenges and gaps? 16. Based upon the cost of the program and the estimated impact, what would be the recommendation for future planning of similar programs? What program areas need more financial resource allocation and why? B. MONITORING AND EVALUATION 17. Have the data gathering methods and indicators used by the project been appropriate for monitoring progress and indicators? Explain. 18. Are there any gaps in the data gathering methods and indicators used to measure progress? 19. To what extent has the programme M&E system collected appropriate, timely and accurate information? 20. How have routine data been used for learning and programme decision-making? 21. What are some of the lessons learnt in providing technical support, capacity building and sub-granting on HIV/AIDS to NGO? C. RELEVANCE AND SUSTAINABILITY 22. To what extent has the Programme been relevant to (a) national HIV and AIDS priorities and strategic plans (b) USAID Malawi strategy (c) sub-grantee needs (d) beneficiary community needs; 23. What sustainability measures or exit strategies were planned at all levels of the program? 24. Are the implemented sustainability measures effective? D. STRATEGIC COLLABORATION 25. What linkages have you developed and with which organizations and on which issues do you collaborate on? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 72 26. What has been the extent to which the project facilitated synergy, coordination and information sharing among local organizations and with the national levels structures such as NAC and key ministries? E. WAY FORWARD AND RECOMMENDATIONS 27. If you have been given chance to implement the programme again would you implement it the same way in term of design? How different would it be? 28. What are some of the recommendations for improvement? F. SPECIFIC DATA REQUIREMENTS Data on planned inputs and accomplished outputs (trainings e.g.) as per Log-frame to help which generating a matrix of planned versus achieved inputs and percentage achievement at input level Total programme costs in – to include programme and administrative costs from various cost centres USAID Malawi, Pact Malawi and sub-grantees From M&E system, records or reports, total number of secondary and primary beneficiaries, appropriately disaggregated and specific programme interventions they benefited from Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 73 ANNEX 6: Pact Sub-Partners Stakeholders Questionnaire This questionnaire is addressed to organizations working in partnership with Pact partners. Purpose: To identify areas of collaboration and networking and assess the extent and impact of these collaborations. Respondents: Respondents of this questionnaire are key stakeholders at district level for Pact‘s partners (local/international NGOs, FBOs, CBO, District Assemblies, and Health Centres 1. What synergies, linkages and partnership were developed with (name of organization) in program inception, planning and implementation? 2. How well do you think the activities of (name of organization) addressed specific problems of beneficiaries? 3. How did the collaboration strategies directly or indirectly improve delivery of services in: a) spread of HIV infection in children, youth and adults b) stigma and discrimination among PLWHAs c) social exclusion of PLWHAs d) Nutritional support for OVCS and PLWHAs e) Psycho social support for OVCs and PLWHAs f) Referrals and access to healthcare for PLWHAs 4. How was the participation and involvement of different stakeholders at different levels of the program and how has this helped in the success of the program? 5. Do you think the program was successful? If yes what were success factors? 6. What were some of the challenges faced in your collaboration with this organization? 7. How did you address with the challenges? 8. In your opinion do you think the programme activities of this organization are sustainable? Explain? 9. What lessons were learnt? 10. Stakeholder Matrix(For facilitators only) Name of partner organ. Area of competence/collaboration Level of collaboration Area of coverage Nature of activities Organizational Constraints Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 74 ANNEX 7: Technical Assistance Questionnaire to Pact Sub-partners This questionnaire is addressed to Pact partners which are local NGOs that Pact was supporting. Purpose: To assess the nature, quality and impact of technical assistance provided to recipient organizations by PACT under the Community REACH programme Respondents: Respondents of this questionnaire should be Executive directors, program managers, finance officers and project officers. Name of Organization: Names and positions of person (s) being interviewed: What role did the person being interviewed play in the Community REACH programme? Name of interviewer: Date of interview: 1. What activities did your organization implement under the Pact community REACH programme? 2. What type of technical assistance did you receive from Pact? 3. How was it decided that you needed assistance in that area (for each area identified in 3 above)? 4. How was this assistance provided? (Probe: Mentorship, coaching, workshop/training secessions, exchange programme) 5. How was this assistance structured? (probe for each of the methods identified under 4 above) 6. How were the results of each of the assistance provided assessed? Were there regular feedback and follow-up sessions? 7. In what ways has your organization benefited from the technical assistance received during this programme? 8. How has this technical assistance improved implementation of your activities? Please explain. 9. What were some of the challenges experienced in relation to the technical assistance? (probe for each challenge identified: what was it, how was it addressed, when was it addressed? 10. During implementation of programme activities, are there areas that you feel needed technical support but the programme did not cover? What are these areas? Why they were not covered? 11. What are some of the lessons learnt? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 75 ANNEX 8: Purpose and Scope of The Evaluation A. PURPOSE OF THE EVALUATION The purpose of this evaluation is threefold; to: 1. Review, analyze and evaluate the extent to which the Community REACH project has achieved its stated objectives. 2. Assess PACT‘s capacity building model‘s success in providing technical assistance and enabling local NGOs, CBOs, and FBOs to grow as organizations, manage larger programs and access other additional funding. 3. Provide specific recommendations and lessons learned that the Mission can explore and utilize in designing future capacity building programs. B. SCOPE OF WORK The contractor shall review, assess and provide documented qualitative and quantitative findings and recommendations of the Community REACH project in Malawi covering the period 2007 to 2011. The evaluation will consider qualitative and quantitative aspects of the program design and results and identify areas in the program that have had the most and the least improvements, reasons why and resulting effects on capacity building and HIV programming in Malawi. The recommendations provided from the evaluation shall be within the context of the current US Mission/PEPFAR and USAID strategic planning or seek to improve upon such. Recommendations shall also reflect Malawi Government strategic planning around HIV/AIDS programming and where possible address gender and sustainability issues. The contractor shall complete the following tasks: 1. Review key background and other related documents 2. Conduct in brief and debrief meetings with USAID/Malawi 3. Interview key informants for the assessment, including sampled local grantees under the project, government officials at all levels of governance, INGO partners working with Pact Malawi staff. 4. Conduct site visits to local sub partners and other relevant program sites to verify activities 5. Based on findings, produce a final evaluation report for the Community REACH project The final evaluation report shall include the following sections: a. USAID branded cover page b. Executive summary, concisely summarizing critical elements of the main report c. Table of contents d. Introduction, describing the purpose and objectives of the evaluation e. Background of the project f. Findings g. Conclusions and lessons learned h. Recommendations for future programming for USAID/Malawi i. Other information relevant to the evaluation but not necessarily central to it may be included in annex, including list of reviewed documents, photos, success stories, interview transcripts, etc. C. METHODOLOGY AND STRATEGY The evaluation shall assess and analyze the following: 1. The effectiveness and efficiency of the project‘s organizational structures in administering sub￾grants, providing technical assistance, building capacity and developing sustainable organizations that can mobilize resources independently, manage resources better and implement strong technical programs? a. To what extent has the project achieved its objectives related to HIV/AIDS technical support, sub￾granting to local NGOs and building capacity of local NGOs? Pact Community REACH Program Final Evaluation Report By Salephera Consulting Ltd Page 76 b. What has been the effectiveness of the sub-granting process to local organizations, the technical capacity provided to local organizations and the capacity building technical assistance provided to local organizations? c. Are there any demonstrable improvements in capacity development and technical performance in the sub recipients? d. What progress has been made in implementing the benchmarked activities negotiated in the agreement and annual work plans and in achieving yearly targets? e. Have the data gathering methods and indicators used by the project been reasonable for monitoring progress and indicators? f. How do sub-grant recipients rate effectiveness of the sub-granting and capacity building support of Pact? 2. The estimated cost effectiveness of the various interventions implemented by Community REACH project relative to the number of beneficiaries reached and the long term impact of the capacity building efforts? Based upon the cost and estimated impact, what would be the recommendation for future planning of similar programs? a. Have sub partner organizations improved their technical and organizational capacity levels to the extent they can be funded directly by USAID? b. What level of population based coverage has the project achieved with its interventions? c. Did the project activities reach a high proportion of beneficiaries of various subgroups such as women, men, young people, PLHIV and OVCs? 3. To what extent has the project achieved its objectives in relation to the national HIV strategic plans? a. What has been the extent to which the project facilitated synergy, coordination and information sharing among local organizations and with the national level structures such as NAC and key Ministries? 4. To what extent was gender integrated in the program? And to what extent was it effective?