December 2012 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Mildred Howard, Danae Roumis, Sula Nakanyika-Mahoney, and David Toomey on behalf of Social Impact, Inc. A Global Development Alliance to Combat HIV/AIDS in the Agribusiness and Mining Sectors in Zambia: End of Project Evaluation ii iii Cover Photo: Zambia Sugar’s Nakambala Estate in Mazabuka, Zambia (November 2012). Photographed by Danae Roumis. Map Source: CHAMP, 2012 iv A GLOBAL DEVELOPMENT ALLIANCE TO COMBAT HIV/AIDS IN THE AGRIBUSINESS AND MINING SECTORS IN ZAMBIA: END OF PROJECT EVALUATION EXPERIENCE FROM A DIRECT PARTNERSHIP WITH NON-TRADITIONAL DEVELOPMENT ACTORS WORKING COOPERATIVELY TO COMBAT HIV/AIDS IN THE WORKPLACE AND COMMUNITY December 2012 Task Order: RAN-1-00-09-00019/RFTOP# SOL-611-12-000010 This report will also be made available on the Development Clearinghouse (DEC). DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. v ACKNOWLEDGMENTS The team wishes to extend their deep thanks to everyone who provided valuable time and information to support this evaluation. Firstly, thank you to our Contracting Officer’s Representative, Patricia Sitimela, at the USAID/Zambia mission who provided valuable support and coordination, as well as helpful technical feedback, especially during the initial stages of this evaluation. The team would also like to thank others at the USAID/Zambia mission, including Mission Director, Dr. Susan K. Brems, and her staff, as well as the USAID/Washington Bureau for Policy, Planning and Learning, Office of Learning Evaluation and Research (PPL/LER), the Program Cycle Service Center (PCSC), and the Africa Bureau, for their guidance and direction on this evaluation as well as their thorough and helpful feedback on our draft report. We would also like to express our appreciation for the generosity and coordination of the local implementing partner, the Comprehensive HIV/AIDS Management Program (CHAMP). CHAMP welcomed our team and often provided workspace in their offices, helped to arrange appointments with stakeholders, and facilitated field visits to the agricultural and mining sector partners. We would also like to acknowledge the agribusiness and mining sector corporate partners, who generously welcomed the team and made senior and technical staff available during the data collection phase of this exercise. Last but not least, we would like to thank Sabreen Alikhan and Maryam Hassan at Social Impact for back-office administrative and technical support during all phases of this evaluation, as well as our logistician in Lusaka, Mwangala Yeta. This evaluation would not have been possible without the support and guidance from all of these partners, and the team is grateful for all of their work and contributions. vi CONTENTS ACKNOWLEDGMENTS .................................................................................................................................................................. v ACRONYMS ..................................................................................................................................................................................... viii EXECUTIVE SUMMARY .................................................................................................................................................................. 1 Evaluation Purpose & Evaluation Questions....................................................................................................................... 1 Project Background ................................................................................................................................................................. 2 Evaluation Design, Methods, & Limitations ........................................................................................................................ 2 Findings & Conclusions ........................................................................................................................................................... 3 Recommendations & Lessons Learned for Utilization ..................................................................................................... 6 EVALUATION PURPOSE & QUESTIONS ................................................................................................................................ 11 Evaluation Purpose ................................................................................................................................................................ 11 Evaluation Questions ............................................................................................................................................................. 12 Organization of the Report ................................................................................................................................................. 12 Project Background ............................................................................................................................................................... 12 Development Hypothesis & Results Framework ............................................................................................................ 14 EVALUATION METHODS ........................................................................................................................................................... 15 Constraints & Limitations .................................................................................................................................................... 19 FINDINGS, CONCLUSIONS, RECOMMENDATIONS, AND LESSONS LEARNED ..................................................... 21 1: Strengthening, Expanding, & Sustaining HIV Workplace Programs ............................................................................. 21 Findings ..................................................................................................................................................................................... 21 Conclusions ............................................................................................................................................................................. 25 Recommendations ................................................................................................................................................................. 25 Lessons Learned ..................................................................................................................................................................... 25 2: GDA/PPP Model Effectiveness in Implementing Activities & Achieving Results ....................................................... 25 Findings ..................................................................................................................................................................................... 25 Conclusions ............................................................................................................................................................................. 35 Recommendations ................................................................................................................................................................. 35 Lessons Learned ..................................................................................................................................................................... 35 3: Reducing the Workplace Impact of HIV/AIDS ................................................................................................................. 36 Findings ..................................................................................................................................................................................... 36 Conclusions ............................................................................................................................................................................. 38 Recommendations ................................................................................................................................................................. 38 Lessons Learned ..................................................................................................................................................................... 38 4: Effectiveness of Working with Local Implementing Organizations and Private Sector Partners .......................... 39 Findings ..................................................................................................................................................................................... 39 Conclusions ............................................................................................................................................................................. 43 Recommendations ................................................................................................................................................................. 43 Lessons Learned ..................................................................................................................................................................... 43 5: Added Value of USG Partnership with GDA Partners ................................................................................................... 44 Findings ..................................................................................................................................................................................... 44 Conclusions ............................................................................................................................................................................. 47 Recommendations ................................................................................................................................................................. 47 Lessons Learned ..................................................................................................................................................................... 47 ANNEXES Annex I: Evaluation Statement of Work Annex II: Evaluation Methods and Limitations Annex III: Data Collection Instruments and Transcripts Annex IV: Sources of Information Annex VI: Disclosure of any Conflicts of Interest Annex VI: List of PEPFAR Indicators, Targets Annex VII: Data, Full List of Tables and Figures Annex VIII: List of Differences from Evaluation Report Draft vii TABLES & FIGURES Table 1. Program and Partnership Objectives and Evaluation Questions Table 2. Evaluation Contribution Analysis Approach Table 3. Data Collection Matrix from GDA Partners Table 4. Company-Reported Results by PEPFAR Performance Indicators Table 5. Comparison of Financial Inputs to Results across GDA Partners Figure 1. COMETS Project Development Hypothesis Figure 2. COMETS Project Results Framework Figure 3. Sensitization: LOP Results Against Targets Figure 4. Counseling and Testing: LOP Results against Targets Figure 5. ART Enrollment: LOP Results against Targets Figure 6. Sensitization: LOP Results against Targets, Gender-Disaggregated Figure 7. Counseling & Testing: LOP Result against Targets, Gender-Disaggregated Figure 8. ART Enrollment: LOP Results against Targets, Gender-Disaggregated Figure 9. New ART Enrollees through Mobile Health Units by District, LOP Figure 10. New ART Enrollees through Mobile Health Units by District, LOP, Gender-Disaggregated Figure 11. Sensitization AB LOP Trends Figure 12. Sensitization Other LOP Trends Figure 13. Counseling & Testing LOP Trends Figure 14. Treatment (ART New) Figure 15. Association between Sensitization (AB) and CT Figure 16. Association between CT and ART Figure 17. Employee Retention and Leave for Zambia Sugar, LOP Figure 18. Medical Expenses for Zambia Sugar, LOP viii ACRONYMS AB Abstinence and “Be Faithful” (as in Sensitization AB) ART Antiretroviral Therapy ARV Antiretroviral Drugs BRAT Business Response for Access to Treatment CBA Cost-Benefit Analysis CHAMP Comprehensive HIV/AIDS Management Program COMETS Community Empowerment Through Self-Reliance CT Counseling and Testing DEC Development Experience Clearinghouse FGD Focus Group Discussion FY Fiscal Year GDA Global Development Alliance GRZ Government of the Republic of Zambia JSI John Snow, Inc. KII Key Informant Interview M&E Monitoring and Evaluation MOH Ministry of Health MOU Memorandum of Understanding NAC National AIDS Council NPI New Partnerships Initiative PEPFAR President’s Emergency Plan for AIDS Relief PMTCT Prevention of Mother-to-Child Transmission of HIV/AIDS PPP Public Private Partnership SHARe Support for HIV and AIDS Response STI Sexually Transmitted Infections SOW Statement of Work USG United States Government USAID U.S. Agency for International Development VCT Voluntary Counseling and Testing 1 EXECUTIVE SUMMARY EVALUATION PURPOSE & EVALUATION QUESTIONS In September 2008, USAID/Zambia awarded a cooperative agreement to the Comprehensive HIV/AIDS Management Program (CHAMP), a private, non-profit local organization, to implement the Community Empowerment Through Self-reliance (COMETS) project. CHAMP was established in 2002 to combat the HIV/AIDS epidemic in Zambia. The purpose of the COMETS project was to expand access to comprehensive HIV/AIDS services and to build relevant capacity in rural and underserved communities in Zambia. COMETS was intended to demonstrate the pivotal role that local organizations and public￾private partnerships can play in increasing access to quality HIV/AIDS and STI prevention, care and treatment interventions in workplaces and communities, and in attaining Zambia’s goals of reducing HIV/STI transmission as well as the social and economic burden of HIV/AIDS. The purpose of this evaluation is to objectively determine the extent to which the COMETS project achieved its goals, which were: (1) To support private sector GDA partners in strengthening their HIV workplace programs and expanding programs into the local surrounding communities; (2) To reduce the impact of HIV/AIDS in the agribusiness and the mining sectors by addressing issues of increasing productivity, reducing absenteeism, retention of skilled employees and labor and increased business output. Secondly, the evaluation seeks to assess the effectiveness, efficiency, and added value of the project’s structure, including: (1) The effectiveness and efficiency of the New Partnerships Initiative (NPI) approach of direct funding to non-traditional local implementing partners; (2) The demonstrated value added by use of a GDA model in the COMETS project. The evaluation addresses a complex and interrelated set of Key Questions (KQs) that are directly correlated with the evaluation’s purpose. The KQs as stated in the Evaluation Statement of Work are: 1. To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? 2. How effective has the Global Development Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? 3. What has been the project’s contribution to reducing the impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? 4. To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost-efficiency, and monitoring and evaluation? 5. Besides the USG funding contribution, what has been the added value, if any, of the USG partnership with GDA partners? a. What has been the added value to private companies, GRZ, and local communities? b. What additional benefits have been realized in engaging private companies that already have corporate social responsibility? c. What multiplier effects have the USG-GDA partnerships had within and outside of HIV/AIDS? 2 PROJECT BACKGROUND HIV continues to challenge to economic growth and development in Zambia. The current HIV prevalence among the most productive age group (15 to 49) is currently estimated at 14.3%.1 An estimated 980 thousand people in the country are living with HIV, making Zambia one of the countries in Sub-Saharan Africa worst affected by the HIV and AIDS pandemic.2 Although HIV is acknowledged as a society-wide problem, rates are particularly high along major highways, trading centers, on plantations, and in mining towns. The concentration of HIV and AIDS on plantations and in mining towns presents a special challenge and opportunity. The private sector companies surrounding these communities are the economic engine of growth and vitality for the communities, and the HIV/AIDS problem adversely affects the productivity of the companies. The companies have a strong vested interest in mitigating the socioeconomic burden of HIV/AIDS among their workforce and in their surrounding communities. In recognition of a convergence of interests, between 2004 and 2008, USAID/Zambia established partnerships with a number of private sector companies within the tourism, agribusiness and mining sectors. There was strong corporate commitment to participate in these early iterations of the partnership model, which quickly demonstrated that where services such as counseling and testing (CT) were readily accessible, demand for a wide variety of other HIV/AIDS prevention, treatment and support services increased exponentially. Accordingly, between 2008 and 2012, the COMETS project facilitated the expansion and deepened application of this GDA model and combined it with strategies for public￾private sector collaboration at the community level. Thus, COMETS was initiated and focused on expanding the scope of workplace programming and community outreach, while bringing existing and new partners into the Global Development Alliance. EVALUATION DESIGN, METHODS, & LIMITATIONS This evaluation addressed issues related to impact on beneficiaries, communities, and corporate partners, but could not be considered a classic impact evaluation as defined by USAID evaluation policy3 because of the lack of a comparison group or valid counterfactual. The COMETS project was not implemented such that these methods were possible, since the evaluation was planned and completed entirely ex post. Instead, this evaluation collected evidence of quantitative and qualitative results and trends that occurred among private partners, their workforces, and individuals in the surrounding communities. The approach taken in this evaluation was a non-experimental but systematic and comprehensive review of program outputs and outcomes on beneficiaries. Data analysis and perspectives as reported by evaluation informants were used to infer the impact of the COMETS activities on participants. The methods were chosen and employed in order to take greatest advantage of the available data in order to generate evidence that is as strong as possible given these significant measurement and evaluation challenges. However, it is important to acknowledge that these methods cannot be used in order to make causal inferences about the impact of the COMETS activities in isolation from the rest of the context in which the program was implemented. 1 Zambia Demographic and Health Survey 2007 2 UNAIDS 2009 Country Progress Report; CHAMP/COMETS Evaluation Statement of Work 3 USAID Evaluation Policy (2011) Evaluation: Learning from Experience. USAID, January 2011: Washington, DC. 3 FINDINGS & CONCLUSIONS FINDINGS Many of the private sector partners had already established HIV workplace programs before the COMETS project. These existing activities were strengthened by the COMETS project. Financial and technical resources provided through COMETS enabled partners to train a larger number of peer educators, produce and distribute information/education pamphlets and other materials, and continue condom distribution. In cases where partners had relationships with or actually maintained health facilities, COMETS support was often used to upgrade facilities, purchase equipment, or support personnel carrying out COMETS activities. The COMETS project also enabled partners involved with MHUs to intensify their support, and during the COMETS project a total of seven MHUs visited several rural health centers fortnightly on a rotating basis. All partners visited and interviewed echoed appreciation for the technical support COMETS provided in revising and strengthening their HIV workplace policies. COMETS assisted in aligning them with national policies, as well as integrating gender-specific components and other language against stigma and discrimination. Several new activities within the existing scope of the HIV workplace programs were also initiated. All four of the partners with whom site visits were conducted reported consistently that their participation in the GDA network was essential to strengthen their HIV workplace programs not only because of the financial support, but also because of the opportunity to share best practices within the network at the annual round-tables and frequent technical meetings throughout the year. Companies whose HIV programs are a component of larger health programs appear to be especially well-poised to take advantage of the partnership. They are able to take advantage of personnel time often dedicated solely to implementing health programs. One of the most important aspects of COMETS was the placement of an M&E assistant at each of the private partners’ facilities during the project. This not only enabled the company to focus on implementation and shift away from reporting burden, but also provided them with a continuity of technical support and capacity that could be shifted over time to the permanent company staff, and also provided health program personnel with tools, such as reporting forms and materials, that can be used on an ongoing basis. The COMETS project enabled partners to expand their HIV programs by expanding the services available for dependents and spouses of employees, as well as by conducting extensive community-based outreach. Private company partners consistently and universally stressed the importance of acknowledging that the community is inseparable from the workforce, and therefore should be attended to accordingly. There is compelling evidence to support two main explanations for the unique role of COMETS in HIV workplace programs. The COMETS funding was critical to motivating companies to expand into the community. Also, the joint funding represents the idea that the parties are working together collaboratively to achieve mutually shared goals. Secondly, the companies agreed to targets for sensitization that may not have been achievable through their activities within the workplace or a small surrounding radius only – these targets may have encouraged companies to reach out to a much larger surrounding population. The COMETS project provided both financial impetus, as well as motivation, stemming from the monitoring of targets. COMETS partially enabled partners to sustain their HIV workplace programs; the workplace-focused activities appear to be more sustainable than community-based activities. However, there did not appear to be an effective strategy to ensure continued funding, support, or transfer of certain key components of the COMETS program. Notably, sentiments regarding the sustainability successes and challenges were consistently expressed by high performing and low performing partners alike. 4 In many cases, partners reported that an important component of the sustainability of their programs was being able to demonstrate their ability to manage such donor-funded projects and participate in cross-sector partnerships. Conversely, COMETS performed to a lesser extent in enabling companies to sustain newly initiated activities after the end of the COMETS project. According to all partners visited, a large portion of the resources dedicated to M&E was not maintained through company budgets after the end of COMETS. The issue of sustainability with regard to community-based peer educators was also distinct. Peer educators recognize that it may not be financially sustainable for private partners or implementing partners to remunerate them in the long-term, or offer ongoing professional development, but many of them expressed the desire to be brought into the planning of programs. All partners visited mentioned the need to integrate senior leadership into the program more closely as well to support sustainability. Senior management was said to have provided their full verbal support, and their support for the programs, but often do not themselves participate in activities that would set an example for the rest of the workforce. Of the total 35 PEPFAR indicators relevant to COMETS, most partners agreed to meet targets in at least 30 of the 35 indicators. Despite the agreed targets, most companies only reported results in 10-13 indicators. The indicators in which company partners reported greatest results are those with the most direct impact on the workforce (treatment) or those that can be provided quickly and easily (sensitization and testing). Private sector partners appeared to be incentivized to impact their employees and surrounding communities in this way by setting targets, and conversely, not incentivized properly over time to re-set targets and to achieve gender balance given the lack of dynamic and gender-oriented target-setting process. Mining companies appeared to be more successful targeting men and agricultural companies were generally more successful with women, while MHUs enrolled more women than men on ART. Many companies cited improvements in productivity indicators – including absenteeism, productivity, and retention, as well as reductions in medical claims and funeral costs – as initial impetus for implementing workplace HIV programs. Actual data was only obtained from Zambia Sugar. The only notable trend is a consistently decreasing turnover rate over the life of the COMETS program. CHAMP is currently in the process of repeating a very detailed effort to conduct a Cost-Benefit Analysis that includes granular data on these indicators that will be compared to a baseline CBA conducted before the start of the COMETS program. At the time of the evaluation, this information was not yet available. In terms of the effect of increasing CT access and ART enrollment, Human Resources professionals that were interviewed suggested that absenteeism, most notably chronic and prolonged absenteeism, had been reduced over the last few years. There is no reason to doubt whether advances in ART delivery, technology, and adherence has been a factor in this reported trend, but the exact contribution of the COMETS program is difficult to isolate. Nevertheless, it is also important to acknowledge the established link between antiretroviral treatment and recoveries of health and the potential for productive work and activity. It is reasonable to look at the achievements that occurred within the life of the COMETS project in terms of counseling and testing and antiretroviral therapy initiation, and conjecture that quality of life will have been increased discernibly over the period between 2008 and 2012 given the large amount of individuals put on treatment. This kind of conjecture is highly dependent on some strong assumptions, of course, including that these individuals were put on treatment early enough to recover to a productive health state, that they continue to have access to treatment, that they adhere to drugs, and otherwise maintain healthy behaviors key to positive living such as good nutrition. There is substantial qualitative evidence that COMETS harnessed robust corporate involvement in workplace programming and influenced and coordinated HIV programming. Specifically, partners reported increased ability to achieve company goals more efficiently as a result of direct cooperation and involvement of USAID. 5 In addition, COMETS provides a model for new private sector entrants into partnerships for workplace health programming. COMETS also provided a useful model for building new programs upon the successes of predecessors. It was clear that partners who had established HIV or other health programs were prepared to implement immediately upon program initiation. CHAMP feels that the partnerships have been a success overall. Despite the fact that some of partners did not meet their targeted financial contributions, the total financial contributions achieved by all partners exceeded total commitments. With regard to the implementation of COMETS, CHAMP reported two major challenges, both related to personnel and the limitations of resources available to CHAMP as a local implementing partner. First, job insecurity for employees caused staffing instability, especially towards the end of the COMETS project, when many people left to join other organizations offering long-term employment prospects. Substantial challenges with regard to procurement were reported by partners, who expressed concerns about delays in acquiring equipment due to CHAMP’s high staff turnover rate. A significant challenge in working directly with local implementing organizations, as reported by most partners, was persistent lack of communication between partners and CHAMP representatives. While visibly well-structured, the CHAMP/COMETS M&E system was not found to be operated efficiently. Overall, local communities were found to have benefited substantially from COMETS as a result of expanded private sector programming. Specifically, working directly in the community was cited as a means of reducing stigma, tracking patients more efficiently, and sharing program ownership. Several partners expressed appreciation at the opportunity that participation in the COMETS program provided, with respect to allowing them to now think about expanding their programs more holistically to include other health areas, including malaria and maternal and child health most prominently. COMETS was seen by most private sector entities as a valuable addition of HIV/AIDS education and services that were complementary to existing workforce programming. Corporate partners were able to reach out to their surrounding communities in a way that may not have been otherwise supported by the company’s budget, regardless of the availability of funds. It is clear that companies have moved to embrace HIV programs in a more entrenched way within the workplace over the life of the project. Not surprisingly, perhaps, the three GDA partners with the highest percentage of USG input also had the lowest percentage of high performing results against indicators. There does not appear to be a positive correlation between degree of USG input and success, but there is some indication that larger total input leads to greater overall success. There appears to be a direct correlation between high performance and the percent of company input into workplace programming. However, the predominance of company contributions in work place programming means that the attribution of USG financial is difficult. CONCLUSIONS COMETS considerably strengthened HIV workplace programs. COMETS expanded HIV workplace programs, especially with respect to community outreach. COMETS partially enabled partners to sustain their HIV workplace programs; workplace-focused activities appear to be more sustainable than community-based activities. Although some constraints were identified by GDA partners, the COMETS project was overall highly effective in implementing project activities. The presence, technical assistance and facilitation of the COMETS project served as a catalyst and provided measurable momentum to GDA workplace programs. The GDA model was effective as a whole in terms of achieving results related to sensitization, counseling and testing, and ART enrollment, though to a lesser degree in the latter category. However, results were driven by a set of high performers and per-partner results were less consistent and did not convey a sense that COMETS participation alone contributed to the success of results achieved. More than simply having long-standing workplace programs, higher-level corporate interest and budgetary commitments were probably reinforced through participation in COMETS. 6 The workplace burden of HIV was reduced through COMETS as a continuation of other programs that predated it. This conclusion is overwhelmingly supported by qualitative evidence, with limited quantitative support provided by partners during the evaluation, though a cost-benefit analysis (CBA) collecting more granular productivity data by CHAMP is ongoing. Overall, USAID’s approach of working directly with local implementing partners has been effective in terms of promoting the achievement of workforce HIV programming goals and increasing stakeholder buy-in and accountability to overarching program objectives. Management of the COMETS project by CHAMP was overall successful, with respect to fostering and maintaining relationships with private sector partners. Chronically poor communication, however, presents a significant challenge to cooperation within partnerships. Other significant challenges were faced in the areas of staff retention, monitoring and evaluation, and procurement procedures. Procurement was not nearly as efficient as private partners would have been satisfied with, which influences future opportunities to continue these partnerships. Monitoring and evaluation professionals were not valued highly enough to be made a permanent part of private sector partner staff post-COMETS. USAID’s cooperative approach was particularly strong in supporting GRZ national health plans by participating in district health planning and management meetings, and by providing financial and coordinative support for running of mobile health units, thereby expanding access to HIV/AIDS services to rural areas. However, due to lack of GRZ capacity, CHAMP/COMETS experienced difficulty transferring ownership of MHUs to the GRZ. The most profound value of the USG partnership with GDAs, as implemented in COMETS, was the ability to build upon prior private sector engagement with their workforce for health interventions in HIV AIDS planning. The COMETS model added the technical competence of CHAMP and its community-based HIV AIDS interventions to the existing USAID experience of working with the private sector to achieve results. Another value addition of the USG/GDA partnership model included partnership with some of the largest industries in Zambia; successful partnering between GRZ and private sector entities increases the credibility of PPP interventions as a model for catalyzing private sector engagement, community outreach and HIV/AIDS program expansion. The intent by GDA partners to attract newer, small enterprises and share knowledge between established and newer partners adds further value to the PPP model, enabling GDAs to fuel the development of technical capacity and small enterprise growth. The USG/GDA partnership has added value to local communities as a result of providing private companies the means to scale up community programming, specifically into areas such as malaria, maternal and child health, and women’s empowerment. COMETS was successful in creating synergies with religious, sports, social and labor organizations for community HIV awareness programming. Because of the size of private sector commitment, often as part of even larger workforce health care delivery, company HIV/AIDS programming (and results) was often specific and focused, emphasizing short-term interventions that directly contributed to a healthy workforce. Degree of USG input does not necessarily translate steer company success, but there is some indication that larger total USG financial input leads to greater overall success. Company success is a key determinant of financial commitment to corporate social responsibility, as demonstrated by a positive correlation between high company performance and the percent of company input into workplace programming. RECOMMENDATIONS & LESSONS LEARNED FOR UTILIZATION USAID has implemented the Global Development Alliance model of Public-Private Partnership (PPP) in Zambia through a local implementer, CHAMP, and a network of private sector companies in the mining and agriculture sectors, aimed at providing services to reduce the burden of HIV/AIDS in the country. Based on the analysis, the evaluation team has concluded that, overall, the GDA/PPP model represents a valid approach toward this goal. 7 When implemented effectively, the GDA/PPP model facilitates communication among private sector companies, coordination between the private sector and the national HIV/AIDS strategic policies, and expansion of services to typically difficult-to-reach populations. By leveraging the joint financial resources of the USG and private sector companies, and the coordination and technical support of the implementing partner, the GDA/PPP model has established a way forward for the public, private, and non-governmental sectors in Zambia to work together toward fulfilling their HIV/AIDS policy objectives. The evaluation team believes strongly that the use of multiple sources and types of data contributes to the validity of the findings presented here, given efforts to rectify any constraints faced. The findings and conclusions present reliable responses to the key evaluation questions, and recommendations and lessons learned respond directly to the intended uses of the evaluation. Although the approach of forming partnerships will differ depending on the local context, many of the conclusions will be generalizable in settings with similar private sector landscapes, burden of HIV, and national health priorities. A summary of the lessons learned and action-oriented recommendations is presented below, in order to assist in the optimal use of the evaluation’s findings, for each intended evaluation user. USAID/Zambia: To identify promising practices and areas for improvement and to inform the design of potential future HIV interventions under USAID/Zambia’s Country Development Cooperation Strategy  Senior management and employees should all be engaged more intensively in the design and planning of HIV workplace programs. These parties provide important inputs that improve the supply and demand of workplace HIV services. Participatory planning may contribute to program sustainability.  In instances where a local implementing partner is providing inputs (such as MHUs) that imply the host government will eventually absorb recurrent costs of said inputs, ensure that an appropriate MOU is in place. Every effort should be made early in the project to ensure that an MOU with the government includes the requirement for government absorption of the item into its budget prior to the end of the project.  The design of future HIV interventions should take into consideration the demonstrated success of the USG/GDA partnership model as a means of promoting private sector engagement in socially responsible activities. Substantial financial input can influence the degree to which private sector partners are able to reach their programming objectives. Also, higher performing partners will invest more in workplace HIV programming, so there may be value to incentivizing high company performance.  Future USG/GDA models should actively promote the expansion of workforce programming into priority areas of the community, rather than focus on short-term interventions directly related to workforce health. Since GDAs already demonstrate they are willing and able to expand programming into local communities, the addition of goal-related incentives may provide further encouragement. USAID/Global Health Bureau, PEPFAR, and other partners in health: to demonstrate effectiveness of the partnership approach with the private sector and local organizations in HIV programs  Future models should consider a diversity of partners, including third-party monitoring and evaluation firms, financial intermediaries, or other relevant stakeholder organizations such as Human Resource professional associations. The COMETS 8 experience has shown that a local non-traditional implementing partner such as CHAMP can serve as a catalyst and effectively provide opportunities for GDA partners in various sectors to learn from each other’s workplace and community outreach programs experiences. The USG/GDA model should be sustained in order to maximize private sector engagement and improve community programming, while focusing greater attention on the priorities and specific strengths of each set of private sector partners in context.  Private sector partners should be encouraged, and given technical support, to formulate HIV workplace program sustainability plans. These plans will highlight their ability to handle donor-funded, multi-sector partnerships. This process may help partners position themselves to attract further funding from a diverse set of donors, to reduce dependency on USG funding. Association and identification with a strategic project such as CHAMP/COMETS provided leverage to GDA implementers to garner internal corporate attention and support to expand their workplace and community activities.  The GDA network’s compulsory technical meetings should continue to be implemented in future replications of the program. Sharing, compiling, and disseminating best practices should become even stronger and more systematic processes within the GDA partnerships. While round-table partner discussions were consistently cited as a major advantage of the project, it appears that there was no systematic effort to document best practices along the way. Future replication of the GDA/PPP model for HIV and other health programs could greatly benefit by a concerted effort to disseminate best practices.  Targets motivate partners to implement activities and achieve results and should be dynamic and gender-oriented in future programs. The COMETS approach of establishing performance targets and accountability memoranda mechanisms, as well as supporting GDAs in being able to measure and report achievements strengthened the GDA/PPP model. This working arrangement between CHAMP/COMETS and GDAs incentivized partners to reach out to their communities to provide services outside of their core businesses, and to reach larger segments of the surrounding population than they might otherwise have.  Providing financial support for HIV programs can add additional value to community health initiatives beyond the realm of HIV. The USG/GDA partnership shows that private sector companies, given the capacity to improve workplace HIV programming, are capable of extending financial benefits to the community to support health initiatives in other areas. USAID/Policy Planning and Learning Bureau: to suggest a model for evaluating public￾private partnerships  Formulate targets that align with GDA goals and project objectives, integrate them into reporting processes, and properly train partners to report accordingly. The reliance on a system of indicators that focuses on only one aspect of the program will not incentivize partners to retain all records in a standardized format that is needed for rigorous assessment of project performance and impact.  Anything that the program intends to measure should be included in the set of targets to be measured before, during, and after the implementation of the project. This includes areas of potential added-value or unplanned effects as much as possible, along with the more obvious workplace impacts (absenteeism, productivity, retention). It is much more difficult and methodologically sub-optimal to collect such measures ex-post. Even where a third-party (such as 9 CHAMP), conducts studies before and after (CBA), it would strengthen the overall evaluation to have the private sector partners tracking such data throughout the duration of the program.  Monitoring and evaluation is a process that cannot exist in a vacuum without any connection to the context in which such data is generated and collected. Methods of incentivizing data collection should conform to the needs of program staff (e.g. bicycles), and attempt to account for unanticipated externalities (e.g. the need for bicycle repair). The ability to track program inputs and outputs in real-time is of critical importance to forming the basis of future programming decisions; therefore, providing appropriate human and technical resources to undertake routine M&E should be prioritized. Companies should be willing to hire an additional monitoring and evaluation professional if they desire to continue tracking progress on key indicators after COMETS program termination, or otherwise explicitly allocate sufficient staff time to carry out these responsibilities. More attention should be placed on how to appropriately incentivize all levels of program staff to routinely record and collect data, with built-in quality assurance processes in place from program initiation through close-out. Government of the Republic of Zambia (GRZ): to demonstrate the effectiveness of the public-private partnership model as a potential methodology for a sustainable HIV response by the government  The GRZ should continue to explore the application of the GDA model for HIV programming, with a view to expanding into other sectors. The GDA model leverages the ability of private partners to reach populations and the ability of other partners to provide the technical, financial, and material resources that need to be distributed.  Invest in Mobile Health Units (MHU) in the recognition of the ability of private sector companies to use their parallel distribution systems to deliver HIV-related care. While national policy currently prioritizes static structures, MHUs appear to be highly successful and should be closely considered as a way to achieve national objectives especially within PPPs. It is clear that private sector partners have the ability to distribute information, materials, and services to wide areas of the population that are otherwise very difficult to reach through the public sector. In addition, they have a vested interest in the health of the communities surrounding their workplace, as the pool from which their workforces are drawn. CHAMP and GDA Partners: to strengthen partnerships and inform future programming  Explore the option of developing a partnership readiness threshold that would serve to “qualify” partners for the program. This would comprise an index of factors that are likely to converge to predispose partners for success, but resources should be leveraged for those with the greatest potential. Then, successful partners can participate in the formulation of strategies to provide newer or more resource-constrained partners with targeted and customized support during project implementation.  Implementing partners should be provided adequate technical and management support, or the resources to obtain it, in order to maximize the potential for success in implementation of project activities and maintain productive partnerships between the implementing partners and private companies. CHAMP expressed initial difficulties with management and procurement, and stretched human resources along with turnover. The fact that private companies and peer educators who felt as though CHAMP was not adequately communicating, or staffed to carry out certain activities, demonstrates the importance of bolstering the capacity of the implementing partner. 10  Resolve issues of sustainability related to the compensation and professional development of community-based peer educators. Peer educators catalyze the process from education to behavior change to healthy living, but occupy a difficult-to-negotiate but opportune space between private companies and communities. All efforts should be made to integrate peer educators into program planning in order to preempt issues of compensation, motivation, and sustainability.  GDA partners who are willing and able to participate in the partnership should be encouraged to compete for government resources, through submission of proposals that include business cases for the PPP, workforce wellness programs, and clear sustainability or exit plans. Companies consistently lacked the ability to quickly produce employee-level data, consistently expressed the desire to apply lessons learned from the HIV programs into broader health programs, and bemoaned the lack of coordination upon project close￾out. Preemptive planning, before partnerships are even established, may improve these issues. 11 EVALUATION PURPOSE & QUESTIONS EVALUATION PURPOSE This evaluation aims to determine the achievements of the COMETS project as well as to assess this application of the USAID Global Development Alliance (GDA) model.4 This evaluation took place between October and December 2012. First, the evaluation seeks to objectively determine the extent to which the COMETS project achieved its goals, which were: (1) To support private sector GDA partners in strengthening their HIV workplace programs and expanding programs into the local surrounding communities; (2) To reduce the impact of HIV/AIDS in the agribusiness and the mining sectors by addressing issues of increasing productivity, reducing absenteeism, retention of skilled employees and labor and increased business output. Secondly, the evaluation seeks to assess the effectiveness, efficiency, and added value of the project’s structure, including: (1) The effectiveness and efficiency of the approach to directly fund non-traditional local implementing partners; (2) The demonstrated value added by the use of the GDA model in the COMETS project. This evaluation is intended to be useful to policy-makers and program managers who have a particular interest in approaches to public-private partnerships (PPPs), including the GDA model, and those with interests relating to the expansion of HIV information, prevention, care and treatment programs. It is anticipated that the main audiences for this evaluation will use the results and lessons learned to inform future programming decisions. This includes, but is not limited to: 1. USAID/Zambia: To identify promising practices and areas for improvement and to inform the design of potential future HIV interventions under USAID/Zambia’s Country Development Cooperation Strategy 2. USAID/Global Health Bureau, PEPFAR, and other partners in health: to demonstrate effectiveness of the partnership approach with the private sector and local organizations in HIV programs 3. USAID/Policy Planning and Learning Bureau: to suggest a model for evaluating public-private partnerships. 4. Government of the Republic of Zambia (GRZ): to demonstrate the effectiveness of the public￾private partnership model as a potential methodology for a sustainable HIV response by the government 5. CHAMP and GDA Partners: to strengthen partnerships and inform future programming It is expected that USAID/Zambia will disseminate the report widely, including in the Development Experience Clearinghouse (DEC), with relevant stakeholders and project beneficiaries, and may use the evaluation in the design of future projects and publications. 4 Evaluation Statement of work is detailed in Annex 1 12 EVALUATION QUESTIONS The evaluation addresses an interrelated set of Key Questions (KQs) that correspond with the evaluation’s purpose: 1. To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? 2. How effective has the Global Development Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? 3. What has been the project’s contribution to reducing the impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? 4. To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost-efficiency, and monitoring and evaluation? 5. Besides the USG funding contribution, what has been the added value, if any, of the USG partnership with GDA partners? a. What has been the added value to private companies, GRZ, and local communities? b. What additional benefits have been realized in engaging private companies that already have corporate social responsibility? c. What multiplier effects have the USG-GDA partnerships had within and outside of HIV/AIDS? ORGANIZATION OF THE REPORT This report presents the evaluation’s findings, conclusions, recommendations, and lessons learned regarding COMETS and the project’s particular application of USAID’s GDA model in Zambia, according to key evaluation questions. Recommendations and lessons learned are also directly linked to the intended uses of the evaluation. Data, tables, and figures, along with interview transcripts, documents reviewed, and other materials are included in the Annex. PROJECT BACKGROUND HIV remains one of the biggest challenges to economic growth and development in Zambia. The current HIV prevalence among the most productive age group (15 to 49) is currently estimated at 14.3%.5 An estimated 980 thousand people in the country are living with HIV, making Zambia one of the countries in Sub-Saharan Africa worst affected by the HIV and AIDS epidemic.6 Although the effects of HIV/AIDS permeate the whole of society, the burden is particularly concentrated high along major highways, trading centers, on plantations and in mining towns.7 This presents a special challenge and opportunity. The private sector companies surrounding these communities are the economic engine of growth and vitality for the communities, and the effects of HIV/AIDS adversely impact the productivity of the companies. The companies therefore have a strong vested interest in mitigating the socioeconomic burden of HIV/AIDS among their workforce and in their surrounding communities. GRZ also has particular interest in targeting areas with the highest infection rates. In recognition of a convergence of interests, between 2004 and 2008, USAID/Zambia established partnerships with a number of private sector companies within the tourism, agribusiness and mining sectors. These partnerships leveraged millions of dollars of resources, expanded HIV/AIDS prevention, 5 Zambia Demographic and Health Survey 2007 6 UNAIDS 2009 Country Progress Report; End-of-Project Evaluation Statement of Work (Contained in Annex I) 7 UNAIDS 2008 Country Progress Report 13 care and treatment services through the private sector and contributed significantly toward achieving PEPFAR targets. From this early partnership it became evident that where services such as Counseling and Testing were readily accessible, demand for a wide variety of other HIV/AIDS prevention, treatment and support services increased significantly. In September 2008, USAID/Zambia awarded a cooperative agreement to the Comprehensive HIV/AIDS Management Program (CHAMP), a private, non-profit local organization, to implement the Community Empowerment Through Self-reliance (COMETS) project. CHAMP was established in 2002 to combat the HIV/AIDS epidemic in Zambia. The purpose of the COMETS project was to expand access to comprehensive HIV/AIDS services and to build relevant capacity in rural and underserved communities in Zambia.8 COMETS was intended to demonstrate the pivotal role that local organizations and public￾private partnerships (PPPs) can play in attaining Zambia’s goals of increasing access to quality HIV/AIDS and STI prevention, care and treatment interventions in workplaces and communities, and in of reducing HIV/STI transmission as well as the social and economic burden of HIV/AIDS.9 The COMETS project began in September 2008 and was completed in September 2012; the last year consisted of a no-cost extension to the original three-year project. The life-of-project cooperative agreement to CHAMP for the COMETS project was in the amount of 9 million USD. COMETS was awarded through USAID’s New Partners Initiative (NPI), a special grant mechanism which allowed for direct funding to non-traditional local implementing partners, and also provided customized capacity building technical assistance to recipient organizations to strengthen their managerial and programmatic effectiveness.10 The COMETS project was implemented with USAID’s innovative Global Development Alliance (GDA) approach to public-private partnerships (PPPs), focused on corporate partners in the agricultural and mining sectors. USAID’s GDA model aims to enhance the impact of development assistance by linking US foreign assistance with the resources and experiences of private sector partners.11 In the GDA model, each partner contributes its own set of skills and resources to collaborate on “co-designed, co-funded, and co-managed” projects and achieve shared objectives.12 In proposing COMETS, CHAMP originally viewed its role as providing or facilitating HIV/AIDS messaging, prevention and treatment interventions in communities surrounding large agricultural and mining entities in Zambia. To align new workplace programming with the relative success of prior workplace programming and to scale the award to a level that CHAMP could successfully absorb, the original grant was revised from USD 13 million to USD 9 million and refocused to incorporate both workplace HIV/AIDS programming in selected companies and community engagement. Accordingly, between 2008 and 2012, the COMETS project facilitated the expansion of the GDA model and combined it with an enhanced strategic scope for public-private sector collaboration at the community level, aligning with PEPFAR indicators and bringing new partners into the Global Development Alliance. 8 USAID/Zambia Cooperative Agreement No. 611-A-00-08-00009-00 dated September, 19, 2008. 9 “Partnership Framework to Support Implementation of the Zambian National Response to HIV and AIDS from 2011 – 2015”, http://www.pepfar.gov/documents/organization/153133.pdf, 2011. 10 Baldwin, Elizabeth, “NPI: Empowering Civil Society, Faith-based Groups, and Communities to Fight HIV, USAID Office of HIV/AIDS, 2012. 11 Lawson, Marian, “Foreign Assistance: Public-Private Partnerships (PPPs)”, 2011. USAID defines GDA as partnerships that strictly encompass the following characteristics: “1:1 leverage of USAID resources, a nontraditional resource partner, a jointly defined solution to a social or economic development problem, shared risks and results, and sustainability.” 12 “Partnering for Impact: PPPs and USAID’s Global Development Alliance Approach”, http://idea.usaid.gov/sites/default/files/attachments/PPPs_vs_GDAs.pdf, 2009. 14 DEVELOPMENT HYPOTHESIS & RESULTS FRAMEWORK The COMETS project was implemented based on a logic framework of activities, inputs, outputs and outcomes13. It is postulated that if adequate resources and level of effort are invested in effectively increasing both demand and supply, health seeking behaviors will improve, health and quality of life outcomes will improve, and the productivity of the community and the workplace will improve. This logic is depicted in the diagram shown below. Figure 1. COMETS Project Development Hypothesis The development hypotheses of the GDA model and COMETS project assume that: (1) Financial inputs would be adequate to support the human and material resources needed; (2) Required systems and competencies would be in place; (3) Demand for services would have been created through effective behavior change strategies; (4) Supply of services would have been adequate to meet demand; (5) An effective monitoring and evaluation system would ensure continuous program strengthening; (6) Opportunities would be seized to leverage and sustain financial, human and material resources. As gleaned from various project documents, the CHAMP/COMETS project’s Results Framework describing purpose, goals, and intended outcomes (results) are shown below. Figure 2. COMETS Project Results Framework 13 This logic framework is detailed in the CHAMP/COMETS FY2012 Monitoring and Evaluation Report Adequate resources and program inputs lead to... Increased supply and demand for interventions and services that lead to... Improved health status and productivity Purpose: To provide support for a program to build capacities of local communities in rural and underserved provinces to deliver comprehensive HIV/AIDS programs in Zambia Goals: To support agribusiness and mining sector GDA partners in strengthening their HIV workplace programs and expanding programs into the local surrounding communities. To reduce the business impact of HIV/AIDS in the agribusiness and the mining sectors. Result #1: Increased knowledge and use of abstinence as a method of preventing HIV Result #2: Increased knowledge and use of alternative methods of preventing HIV infection Result #3: Improved quality of life of PLWHAs and OVCs through care, support and treatment services Result #4: Increased access to services for pregnant women including TC, PMTCT, and ARV prophylaxis Result #5: Strengthened capacity of CHAMP and COMETS partners to collect and use information to strengthen systems and expand HIV interventions 15 EVALUATION METHODS A detailed evaluation methodology, including strategies and limitations, may be found in Annex II. Despite the focus on the achievements and impacts of the GDA model and COMETS program, it is important to note that this evaluation cannot be considered a classic impact evaluation per the USAID Evaluation Policy. True impact evaluations pursue the ideal experimental design with a valid counterfactual, allowing the evaluator to make causal claims attributing observed changes in the outcomes of interest directly to the project or program in question.14 This evaluation was planned and completed entirely ex post, lacking credible baseline measures and valid comparison groups. However, the program objectives clearly reflect the attempt to improve outcomes of interest directly through the COMETS project, and key evaluation questions are oriented toward the specific contribution of the COMETS project to the outcomes of interest (Table 1). Table 1. Program and Partnership Objectives and Evaluation Questions Overall GDA Goals COMETS Specific Objectives Key Evaluation Questions To enhance the GDA partner workplace programs and expand them into the outreach community. 1. Increasing knowledge and promoting behaviour changes by reaching individuals through activities that promote Prevention AB. 2. Increasing knowledge and promoting behaviour changes by reaching individuals through activities that promote Prevention Other 3. Improving the quality of life of People Living with HIV/AIDS (PLWHA) and Orphans and Vulnerable Children (OVC) through care, support and treatment services. 4. Increasing access to testing and counselling (TC) including the Provision of Prevention of Mother-to-Child Transmission (PMTCT) services to pregnant women through activities that include Testing and Counselling and Antiretroviral (ARV) prophylaxis. 5. Strengthening the capacity of CHAMP and Public Private Partners (PPP) to strategically collect and use information and strengthen systems to implement multi-pronged HIV Programmes in support of the project and the national HIV response. KQ 1: To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? KQ4: To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost￾efficiency, and monitoring and evaluation? KQ5: Besides the USG funding contribution, what has been the added value, if any, of the USG partnership with GDA partners? (a) What has been the added value to private companies, GRZ, and local communities? (b) What additional benefits have been realized in engaging private companies that already have corporate social responsibility? To reduce the impact of HIV and AIDS in the agribusiness and the mining sectors by addressing issues of increasing productivity, reducing absenteeism, retention of skilled employees and labor and increased business output KQ 2: How effective has the Global Development Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? KQ 3: What has been the project’s contribution to reducing the impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? The team instead attempted to uncover causal pathways through non-experimental, mixed methodologies. One framework that has been used to describe our implicit approach is contribution analysis.15 Contribution analysis tests proposed associations, mechanisms, and underlying assumptions between activities and outcomes with a view to uncovering the unique contribution of the project as well as the credibility of alternative explanations for any observed effects. Throughout this process, generalizability of findings is also determined. Our approach employed the implicit approach of contribution analysis to identify and consolidate all potential contributions to impact, both due to COMETS and factors unrelated to program activities, summarized in Table 2. 14 USAID Evaluation Policy (2011) Evaluation: Learning from Experience. USAID, January 2011: Washington, DC. 15 Mayne J. 2001. “Addressing attribution through contribution analysis: using performance measures sensibly.” The Canadian Journal of Program Evaluation, Vol. 16, No. 1, pp. 1-24. Kotvojs F, and B Shrimpton. 2007. “Contribution analysis: A new approach to evaluation in international development.” Evaluation Journal of Australasia, Vol. 7, No. 1, pp. 27-35. 16 Table 2. Evaluation Contribution Analysis Approach SERVICE AREA COMETS ACTIVITIES16 MECHANISMS ASSUMPTIONS INTENDED OUTCOMES INTENDED IMPACTS INTENDED LINK TO COMETS ALTERNATIVE EXPLANATIONS DATA COLLECTION METHODS USED SENSITIZATION (EDUCATION) Strategic Objectives 1, 2, and 4 Key Evaluation Question 2 Conduct mobilization and sensitization in target population by HRPN, MHU and CHAMP; Train HRPN on promoting HIV prevention; Distribute condoms to target population; Develop , print and distribute Care Support and Treatment IEC materials Increased education, information, knowledge leads to behavior change Financial inputs would be adequate to support the human and material resources needed Demand for services would have been created through effective behavior change strategies Supply of services would have been adequate to meet demand Increased condom use/STI prevention Reduced multiple concurrent partnerships Increased number of people testing and counseling Reduced transmission of HIV/AIDS Knowledge received through COMETS￾funded sensitization leads to behavior change COMETS provides greater access to condoms, thereby motivating individuals to practice safer sex Knowledge received through other sources led to behavior change Behavior change in this context is affected more sensitively by the availability of relevant resources (e.g. condoms) and the means to act (e.g. money to buy condoms), or other circumstances (e.g. bargaining power in a sexual relationship), rather than the content of the information, education, sensitization provided by COMETS. Document Review Data Abstraction Semi-structured Interviews COUNSELING AND TESTING Strategic Objective 4 Key Evaluation Question 2 Conduct mobilization and sensitization on CT in target population by HRPN, MHU and CHAMP; Train health care professionals and HRPN to conduct opt-out CT Testing will lead to individuals knowing and acknowledging their HIV status. Testing will lead to enrollment on pre￾ART for those HIV positive but not yet eligible for treatment. Testing will lead to enrollment on ART for those eligible for treatment. Financial inputs would be adequate to support the human and material resources needed Required systems and competencies would be in place Demand for services would have been created through effective behavior change strategies Increased number of people know their status; Increased number of people enroll in pre-ART monitoring Increased number of eligible individuals enroll in treatment Reduced transmission of HIV/AIDS Increased quality of life of PLHIV COMETS provides financial and technical support to expand VCT facilities in the workplace, surrounding communities, and remote areas (through MHUs) Increased availability of testing facilities lead to increase in the number of people seeking CT Decision to seek CT influenced by social or economic factors unrelated to COMETS inputs CT received through other sources, rather than those sponsored by COMETS Other initiatives or campaigns are responsible for driving demand for CT Document Review Data Abstraction Semi-structured Interviews TREATMENT AND CARE Strategic Objectives 3 and 4 Key Evaluation Identify and Register Care, Support and Treatment clients; Train HRPN in providing Care and Support and Treatment to HIV infected Increased demand for CT will lead to higher enrollment on ART Enrollment in treatment will Financial inputs would be adequate to support the human and material resources needed Required systems and Increased number of eligible people on treatment Improved productivity Reduced social and economic burden of HIV￾related illness; Increased productive lifespan for COMETS trainings, provision of materials and technical support for HRPN and rural health centers, facilitated an Other governmental or non￾governmental initiatives are responsible for initiatives or campaigns that encourage people to enroll in treatment. Other projects, organizations, Document Review Data Abstraction Semi-structured Interviews 16 A selection of activities listed in CHAMP’s Logic Framework (See CHAMP M&E FY2012 Report) 17 Question 2 individuals; Establish and strengthen a referral procedure between the HRPN and organizations providing care and support; Provide technical support to COMETS partners to scale up CT services; Provide test kits to COMETS partners and HRPN; (CHAMP staff) Participate in USG Palliative Care Forum increase as a result of increased accessibility and availability of accredited facilities and trained personnel competencies would be in place An effective monitoring and evaluation system would ensure continuous program strengthening Opportunities would be seized to leverage and sustain financial, human and material resources indicators (absenteeism, productivity, retention) in the private sector employees with HIV; Reduced stigma in the workplace increase in the number of people on treatment. or initiatives created demand for treatment and care by providing personnel and/or facilities. CAPACITY￾BUILDING AND MODEL EFFECTIVENESS Strategic Objective 5 Key Evaluation Question 1 Conduct training on strategic information Undertake monitoring and evaluation data collection and dissemination Participation in training on monitoring and evaluation systems creates sustainable capacity to maintain reporting requirements within the national HIV framework and within public private partnerships Financial inputs would be adequate to support the human and material resources needed Required systems and competencies would be in place An effective monitoring and evaluation system would ensure continuous program strengthening Opportunities would be seized to leverage and sustain financial, human and material resources Increased capacity of private sector partners to implement, maintain, and report on workplace programs; Improve capacity of private sector partners to coordinate reporting for different streams and to report on indicators aligned with national policy (following the “three ones” principles) Increased private sector engagement in the health of their workforces and surrounding communities Increased private sector accountability and opportunity for learning to inform future programming decisions COMETS-supported training to individuals seconded to private sector partners as M&E Assistants increased the capacity of private sector partners to report based on national guidelines. COMETS-supported training of M&E Assistants increased the capacity of private sector partners to sustain organized, reliable monitoring and evaluation programs for their HIV programs. Training or support from other sources is responsible for any changes in monitoring and evaluation capacity among private sector partners. Document Review Semi-structured Interviews 18 Data Collection Every effort was made to collect data from all 14 partners when possible, specifically through CHAMP/COMETS Monitoring and Evaluation Database (“CHAMP M&E”) and SmartCare systems17, as well as financial records and annual project reports. . A matrix of the data requested and obtained in this evaluation is presented in Table 3. Table 3. Data Collection Matrix from GDA Partners Quantitative Qualitative PARTNER Budget information (Budgets against targets/achievements) Record of COMETS Target Achievements Employee Productivity Data Key Informant Interview: HIV Program Key Personnel Other Zambia Sugar Received Referred to CHAMP Received Completed Focus Group Discussion conducted with Peer Health Educators in the Zambia Sugar catchment Dunavant Received Referred to CHAMP Not Provided Completed First Quantum Mining Not Provided Referred to CHAMP Referred to CHAMP Not Provided Completed Key informant interview with Workplace HIV Peer Educator Mopani Copper Mining Not Provided Referred to CHAMP Referred to CHAMP Not Provided Completed Health indicator targets provided for on-site health facility Zambia Breweries Not attempted * Not attempted Not attempted Completed Not attempted Zambeef Not attempted Not attempted Not attempted Completed Not attempted Albidon Mining Not attempted Not attempted Not attempted Attempts made to contact HRM s Contact non￾functional. Not attempted Kagem Mining Not attempted Not attempted Not attempted Completed Not attempted *Only Key Informant Interviews were attempted with Zambia Breweries, Zambeef, Albidon Mining, and Kagem Mining. Only after site visits with the first 4 partners listed was CHAMP M&E data made available and assessed, highlighting that high￾performing partners had been visited during the in-country phase of data collection. The purpose of site visits was to obtain both quantitative data (e.g. employee productivity and partner financial data), and qualitative data (key informant interviews). Quantitative data used for this evaluation included data from the CHAMP Monitoring and Evaluation database (CHAMP M&E), data extracted from the SmartCare system linked to the Mobile Health Units (MHUs), and financial data from partners. Data was also available from the records of private sector partners, though far more limited in terms of the breadth originally expected. Qualitative data included key informant interviews with health program and HIV-specific personnel at companies, peer educators, and other key informants, including a former COMETS AOR, former GDA chairman, and representatives from FH360, and the GRZ MOH.18 Notes from these sessions and transcribed interviews are included in Annex III. In addition, a Focus Group 17 SmartCare is an electronic health record system in Zambia, used nationally for HIV care and treatment among other areas 18 A list of persons contacted and interviewed is included in Annex IV 19 Discussion (FGD) was organized and conducted at one GDA partner site, Zambia Sugar in Mazabuka, to solicit direct input from peer health educators. Significant document review was undertaken prior to the analysis of any quantitative data, in order to establish a context for the evaluation and the choice of a conceptual framework, understand the key activities undertaken within the COMETS program that would be most influential in achieving the project’s ultimate objectives, and to gain an understanding of any other essential factors of the program as whole in order to usefully contextualize the quantitative analysis and findings in order to make action-oriented recommendations.19 Quantitative analysis was performed using company records from a selection of partners, CHAMP M&E data from all 14 partners, and SmartCare data from the seven Mobile Health Units (MHUs) each visiting multiple Rural Health Centers (RHCs) as part of the COMETS project.20 Using these data, life of project outputs were linked with COMETS strategic objectives. Wherever data was available for both males and females, gender-disaggregation was performed. The team attempted to reconstruct baseline data for productivity areas by requesting information on employee productivity indicators starting from 2007 or 2008 and quarterly over the life of the project, with the earliest data serving as the baseline for these analyses, but faced several constraints which are outlined in other areas of this report, limiting a proper before-and-after analysis. Qualitative methods of data collection and analysis were used both as primary methods of addressing some key questions, as well as in a way to triangulate the findings from other questions that used primarily quantitative data. For example, qualitative methods were used primarily with respect to understanding the management and structural strengths and challenges of the GDA model as in Key Question 4, while they were used mostly to triangulate with respect to project achievements, as in Key Question 2. By using a combination of quantitative and qualitative methods, the team was able to assess the potential mechanisms underpinning the relationships between project activities and intended outcomes, as identified in our evaluation approach. CONSTRAINTS & LIMITATIONS The constraints in this evaluation mainly relate to the availability of certain types of data and the inability to visit all private sector partners. One constraint in carrying out our analysis was related to the availability of data from private sector partners. First, private companies appear to need advance notice beyond what was provided within the timeframe of this evaluation to compile data from over a period of several years. The team views this as a constraint more generally related to the reporting and monitoring and evaluation system of the GDA/PPP, rather than a challenge that was specific to this evaluation. In general, the system relied on PEPFAR indicators to formulate MOUs and track partners’ progress, even though the stated GDA goals were much more broad and included workplace impacts and structural components. Partners were not required to track or report on areas such as productivity and absenteeism as part of the partnership. It was therefore difficult to obtain any data on the important productivity indicators for this analysis (absenteeism, productivity, retention) ex post. In addition, companies purposely do not discriminate against HIV, as a matter of policy. Therefore, absences are not specifically marked as due to HIV/AIDS. Any changes in absenteeism, productivity, and retention therefore could, at best, be correlated with increased activities targeted at HIV/AIDS, but HIV-related absenteeism cannot be truly isolated. The cost-benefit and originally planned statistical analyses in particular were affected by this situation, since both had originally intended to focus on aspects of the impact of the COMETS program on the productivity of corporate partners. 19 A complete list of documents reviewed is included in Annex IV 20 CHAMP M&E Data and SmartCare data are annexed to this evaluation 20 Early in the evaluation, the team acknowledged the proprietary nature of data requested from private companies – specifically, that companies may be keen to preserve sector-wide competitiveness, thereby choosing not to release data on productivity indicators (employee absenteeism, productivity, and retention). However, the evaluation team based its methodology and implementation strategy on the fundamental expectation that such data would be made available to them, as indicated in correspondence with the Mission prior to mobilization. Given the stated goals of the GDA, it appears to that future programs should consider ways to work with Human Resources department to confidentially and appropriately collect such data during the duration of the project. In an attempt to rectify these challenges, the evaluation team specifically met with Human Resources personnel during site visits to assure partners about the intended use and confidentiality of the data. Some partners agreed to release data, while others were reticent or simply lacked the ability or authorization to do so. A similar challenge was met with respect to the financial data needed to fulfill the original intention to conduct cost-benefit analysis (CBA). The financial data obtained from a limited number of partners was not sufficient for a COMETS-wide CBA. While activity-linked expenditure data was theoretically available from CHAMP, as requests for funding from partners during the project had to be linked to specific activity areas with codes, such information and records were not made available to the team during official data requests. It therefore proved difficult for the evaluation team to undertake a traditional cost-benefit analysis due to the misalignment of partners’ budget line items and COMETS indicators. Nevertheless, some of the financial data obtained from partners was still be used for the evaluation outside of the context of CBA. The team attempted to mitigate this limitation by analyzing the relationship between partners’ financial inputs and the achievement of partner-specific activities. The evaluation was also challenged by the need to purposively select of a limited number of GDAs from which to draw conclusions. A sample of four partners was agreed upon during the team’s in-brief with the mission, mainly to balance geographic representation and the time available for data collection. With the intent to capture as much variation in partner programs and experiences as possible, the evaluation team established criteria for the selection of widely representative models, which included sector, program breadth, firm characteristics, program longevity, and geography. In the first meeting with the implementing partner, CHAMP, the team presented their criteria as well as their requests for monitoring and evaluation data. Two mining and two agribusiness partners were suggested to the team for site visits, but the release of M&E data was delayed and therefore, in the interest of time, site visits were arranged prior to the team’s ability to conduct preliminary data analyses. By the time monitoring and evaluation data was obtained after numerous follow-ups, all in-country site visits to partners had already been scheduled. Once M&E data was received, the team determined that the partners visited had been the highest performers. In an attempt to capture data more representative of GDAs as a whole, the team requested from CHAMP the full contact list of all GDA partners in the COMETS project. The evaluation team then addressed this challenge by having its Lusaka-based evaluator continue to reach out specifically to lower-performing partners for interviews after the formal in-country data collection period ended. Information gleaned from those data requests has been successfully incorporated into the findings of this evaluation. The evaluation team faced a potential limitation at the onset of data collection when one team member became unavailable; however, the evaluator was promptly replaced by another Technical Specialist who had initially planned to join the team remotely. Data collection resumed with no delay, and the team continued its evaluation as planned. Even in the presence of some limitations, the data collection activities did yield sources of rich information to answer the evaluation questions, and the team followed the steps delineated in the methodology and approach to accommodate the level of technical rigor expected by the Mission. 21 FINDINGS, CONCLUSIONS, RECOMMENDATIONS, AND LESSONS LEARNED This section presents findings, conclusions, and recommendations about the COMETS program and this application of USAID’s GDA model in Zambia, organized by Key Evaluation Question. Findings include the empirical results synthesized from the data collected while drawing links back to the evaluation framework, with a view to validating evidence for the proposed links between COMETS activities and intended outcomes. Conclusions are drawn directly from the findings, and provide the evaluation team’s responses to the Key Evaluation Questions. Recommendations outline action-oriented items derived from the findings and conclusions. Finally, a set of Lessons Learned for each question responds directly to the intended uses of the evaluation including the design of future HIV programming and GDA models. 1: STRENGTHENING, EXPANDING, & SUSTAINING HIV WORKPLACE PROGRAMS Evaluation Question 1: To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? FINDINGS The evaluation team conceptualized the focus of this question in the following way:  Strengthening: The improvement of existing activities within HIV programs, or initiation of new activities within the existing scope of HIV programs;  Expansion: An increase in the magnitude of their programs’ scope and reach, both within and outside of the workplace; and  Sustainability: Enabling partners to maintain activities that were initiated or improved as part of the COMETS program, through financial, technical, material, and human resources. Strengthening: Many of the private sector partners had already established HIV workplace programs before the COMETS project. The existing scope of HIV programs mainly encompassed the dissemination of HIV information within the workplace, along with services within prevention, counseling and testing, and treatment. Most companies had HIV workplace policies as well before the COMETS program, some considerably more developed than others. Common elements of existing HIV programs include peer educators, information/education campaigns, condom distribution, voluntary counseling and testing (VCT), and referral services to local health facilities. Some partners also had existing networks of community-based peer educators. Others had already been involved in community outreach through the operation of Mobile Health Units (MHUs) providing capacity to Rural Health Centers (RHCs) to provide HIV-related care and treatment which are typically resource-constrained and serve much harder to reach, rural populations. These existing activities were strengthened by the COMETS project. Financial and technical resources provided through COMETS enabled partners to train a larger number of peer educators, produce and distribute information/education pamphlets and other materials, and continue condom distribution. For example, at First Quantum Mining (“First Quantum”), COMETS supported the training of peer educators and provided funding for shirts that identified them within the workplace as resource persons. In total, over 2 thousand workplace and community volunteers were trained under COMETS. Several trainings were held over the life of the project, including Peer Education, psychosocial counselor training, OVC caregiving, Palliative caregiving, Antiretroviral Therapy (ART) and Prevention of Mother￾to-Child Transmission of HIV (PMTCT). Most trainings explicitly incorporated stigma reduction and gender-oriented components. In cases where partners had relationships with or actually maintained 22 health facilities, COMETS support was often used to upgrade facilities, purchase equipment, or support personnel carrying out COMETS activities. Mopani Copper Mines (“Mopani”) maintain a network of health facilities across several sites in northern Zambia in the vicinity of their mines and processing plants. COMETS funding was used to expand a wing of a hospital, support HIV program personnel, and purchase equipment such as a PCR machine, which measures HIV viral load. The COMETS project also enabled partners involved with MHUs to intensify their support, and during the COMETS project a total of seven MHUs visited several rural health centers fortnightly on a rotating basis. All partners visited and interviewed echoed appreciation for the technical support COMETS provided in revising and strengthening their HIV workplace policies. COMETS assisted in aligning them with national policies, as well as integrating gender-specific components and other language against stigma and discrimination that communicated to employees a clear willingness on the part of the company to create a safe environment for all within the workplace. Several new activities within the existing scope of the HIV workplace programs were also initiated. Several partners strengthened their ability to dissemination HIV information in the workplace by initiating employee peer support groups and incorporated HIV talks into the daily activities of the company. For example, mining companies reported integrating HIV topics into daily safety and induction talks. It is notable that both of these activities, and others, were ideas shared between partners in the GDA network. All four of the partners with whom site visits were conducted reported consistently that their participation in the GDA network was essential to strengthen their HIV workplace programs not only because of the financial support but also because of the opportunity to share best practices within the network at the annual round-tables and frequent technical meetings throughout the year. These meetings provided a forum through which partners could discuss their programs with others experiencing similar challenges, even despite the fact that many of these partners worked in the same sector and might otherwise be considered competitors. COMETS, through its ability to bring partners physically to the same table, provided the opportunity for partners to learn from others’ successes and prioritize the health of their workforces and the surrounding communities, which the companies recognize as being inseparable from their workforce. Companies whose HIV programs are a component of larger health programs appear to be especially well poised to take advantage of the partnership. They are able to take advantage of personnel time often dedicated solely to implementing health programs, such as at First Quantum Mining. Accordingly, Dunavant expressed the challenge of being able to manage the administrative and reporting components of participation without more than one or two staff members dedicated full-time to such programs. Along these lines, it was also noted in all site visits that one of the most important aspects of COMETS was the placement of an M&E assistant at each of the private partners’ facilities during the project. This not only enabled the company to focus on implementation and shift away from reporting burden, but also provided them with a continuity of technical support and capacity that could be shifted over time to the permanent company staff, and also provided health program personnel with tools such as reporting forms and materials, that can be used on an ongoing basis. Expansion: The COMETS project enabled partners to expand their HIV programs by expanding the services available for dependents and spouses of employees, as well as by conducting extensive community-based outreach. Private company partners consistently and universally stressed the importance of acknowledging that the community is inseparable from the workforce, and therefore should be attended to accordingly. Two companies (one mining, and one agribusiness), specifically mentioned that the COMETS funding allowed companies to be creative with respect to this expansion; even though they were working toward specific targets, COMETS allowed partners to propose, and upon approval, implement any activities they thought appropriate in order to meet such targets. For example, both Zambia Sugar and First Quantum Mining (FQML) implemented several different types of 23 community outreach programs including attending sporting events, community festivals, and implementing their own small events which captured their target audience through entertainment and social activities, while providing easily accessible VCT services. FQML used COMETS support to produce kitenge, cloth worn by women, printed with HIV prevention messages. They also used existing prevention programs such as “One Man Can” as a formula to develop “One Woman Can” and “One Family Can” programs that involve the entire family unit in HIV and other health promotion activities. Based on extensive interviews with key informants from private sector partners as well as stakeholders from the GDA and the Mission, there is compelling evidence to support two main explanations for the unique role of COMETS in HIV workplace programs. First, although presumably several partners would have had the resources to carry out the activities within COMETS project without USG financial support, the COMETS funding was critical to motivating companies to expand into the community Regardless of the availability of funding, it will always seem to be good business sense to spend on activities outside of the “core business” with money that does not come directly from company revenue. In this way, any support is welcomed by the partners. Also, the joint funding represents the idea that the parties are working together collaboratively to achieve mutually shared goals. For both of these reasons, COMETS provided a platform on which companies could justify their community activities, both internally within company operations and externally – in fact, FQML personnel mentioned the fact that the partnership assisted in providing them with a “social license to operate” in the community. Secondly, the companies agreed to targets for sensitization that may not have been achievable through their activities within the workplace or a small surrounding radius only – these targets may have encouraged companies to reach out to a much larger surrounding population. Indeed, one partner specifically mentioned being very motivated to conduct outreach based on the need to meet the COMETS targets. Another partner expressed that it had always been their intention to expand their activities throughout the entire district, and so the COMETS project provided both financial impetus, as well as the motivation stemming from the monitoring of targets. In other words, the accountability involved in the monitoring and evaluation of targets was also a part of enabling companies to expand their HIV programs. Sustainability: COMETS partially enabled partners to sustain their HIV workplace programs; the workplace-focused activities appear to be more sustainable than community-based activities. However, there did not appear to be an effective strategy to ensure continued funding, support, or transfer of certain key components of the COMETS program. Notably, sentiments regarding the sustainability successes and challenges were consistently expressed by high performing and low performing partners alike. It was also evident within interviews with peer educators trained through COMETS. In terms of the sustainability successes, COMETS enabled companies to continue existing activities through financial and technical support, as well as demonstrate their ability to manage donor-funded activities and serve as effective collaborators in multi-sector partnerships. Partners reported in many cases that they were able to demonstrate their ability to manage such donor-funded projects and participate in cross-sector partnerships was an important component of the sustainability of their programs. For example, one partner reported that they were able to leverage their experience with the COMETS project in order to attract additional technical support from other donors. Bringing private sector partners into the national strategic framework for HIV was beneficial for both parties in terms of sustainability. Private sector partners have incredible distribution capacity – Dunavant even used sheds for seeds to store condoms – while public agencies and non-governmental organizations have the vision and technical capacity to provide policy guidance, medical care, supplies, and other resources to the companies. 24 Conversely, COMETS performed to a lesser extent in enabling companies to sustain newly initiated activities after the end of the COMETS project. In cases where company partners already operated HIV or broader health programs, COMETS funding was used effectively to sustain program activities such as mobile health units, the production of materials for information/education campaigns within the workplace, and support of mobile health units (MHUs). In addition, private sector partners had access through the partnership to the national medical stores, which allowed them to provide ART for their employees at a low cost that would have otherwise been prohibitively high if they were responsible for the full cost of the drugs. Many of those interviewed stressed the importance of the linkages between the private workplace and the national guidelines and medical stores in order to maintain access for their employees to important treatments. According to all partners visited, a large portion of the resources dedicated to M&E was not maintained through company budgets after the end of COMETS. COMETS support for equipment such as bicycles for peer educators, condoms for distribution at sensitization sessions, and physical pamphlets and other materials for use as part of information/education campaigns both within and outside of the workplace were also highly praised by partners, but production and distribution of such materials was apparently scaled back significantly after COMETS instead of being funded by companies or other partners. From a structural point of view, it is possible that no party is responsible for this challenge, but that it was a weakness in the model that a comprehensive exit strategy and sustainability plan was not made compulsory for each GDA with significant input from CHAMP. The issue of sustainability with regard to community-based peer educators was also distinct. Peer educators should be – and often are – regarded as an integral part of health program sustainability. Their sensitization activities serve as the catalyst in the process that starts with education and continues through behavior change, treatment and care, and healthy, positive living. Peer educators consistently expressed the need for a system that links the trainings received through COMETS in a clear, graduated manner rather than participating in a series of one-off sessions. In addition, individuals were trained as trainers of trainees (TOT), and the number of individuals trained in this way was tracked as part of the core set of PEPFAR indicators. Remuneration and continual training of community-based peer educators can present a particular challenge for private companies who value their services but are understandably reticent to dedicate indefinite ongoing financial resources to a cadre of workers not on company payroll for core business activities. However, across sectors and partners interviewed, the team consistently heard that individuals trained as TOTs were never actually given any follow-on opportunity to train others. It was also evident that when COMETS program funding started to end, incentives to peer health educators were reduced or cut entirely, and sensitization efforts therefore could not be maintained in the same way. Peer educators recognize that it may not be financially sustainable for private partners or implementing partners to remunerate them in the long-term, or offer ongoing professional development, but many of them expressed the desire to be brought into the planning of programs in order to be able to discuss the possibility of developing income-generating activities that could assist with sustainability. CHAMP has rightly raised the point that IGAs as an incentive substitute are not “one-size fits all” solutions since not all peer educators want to run or are able to run businesses, and because not all contexts provide sufficient markets for sustainable income generation. Nonetheless, it remains to be determined how to incent and compensate peer educators long-term. All partners visited mentioned the need to integrate senior leadership into the program more closely as well toward sustainability. Senior management was said to have provided their full verbal support, and their support for the programs, but often do not themselves participate in VCT and other activities that would set an example for the rest of the workforce. Lastly, no partners interviewed reported doing any formal survey of employees or community, or holding any type of workshop, in order to directly involve them in the process of planning their HIV workplace program activities. 25 CONCLUSIONS  COMETS considerably strengthened HIV workplace programs.  COMETS expanded HIV workplace programs, especially with respect to community outreach.  COMETS partially enabled partners to sustain their HIV workplace programs; workplace-focused activities appear to be more sustainable than community-based activities. RECOMMENDATIONS  The GDA network’s compulsory technical meetings should continue to be implemented in future replications of the program. Sharing, compiling, and disseminating best practices should become even stronger and more systematic processes within the GDA partnerships.  Private sector partners should be encouraged, and given technical support, to formulate HIV workplace program sustainability plans. These plans will highlight their ability to handle donor￾funded, multi-sector partnerships. This process might help partners position themselves to attract further funding from a diverse set of donors, to reduce dependency on USG funding.  Senior management and employees should all be engaged more intensively in the design and planning of HIV workplace programs. These parties provide important inputs that improve the supply and demand of workplace HIV services. Participatory planning may contribute to program sustainability. LESSONS LEARNED  The COMETS experience has shown that a local non-traditional implementing partner such as CHAMP can serve as a catalyst and effectively provide opportunities for GDA partners in various sectors to learn from each other’s workplace and community outreach programs experiences.  The COMETS approach of establishing performance targets and accountability memoranda mechanisms, as well as supporting GDAs in being able to measure and report achievements strengthened the GDA/PPP model. This working arrangement between CHAMP/COMETS and GDAs incentivized partners to reach out to their communities to provide services outside of their core businesses, and to reach larger segments of the surrounding population than they might otherwise have.  Association and identification with a strategic project such as CHAMP/COMETS provided leverage to GDA implementers to garner internal corporate attention and support to expand their workplace and community activities.  While round-table partner discussions were consistently cited as a major advantage of the project, it appears that there was no systematic effort to document best practices along the way. Future replication of the GDA/PPP model for HIV and other health programs could greatly benefit by a concerted effort to disseminate best practices. 2: GDA/PPP MODEL EFFECTIVENESS IN IMPLEMENTING ACTIVITIES & ACHIEVING RESULTS Evaluation Question 2: How effective has the Global Development Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? FINDINGS To assess the implementation of activities and achievement of results, the evaluators aligned the COMETS targets measured in their monitoring and evaluation system (modeled precisely on pre￾formulated PEPFAR targets) with results reported by company and by CHAMP. Company results were 26 mostly disaggregated by year, and in other cases, cumulative over the Life of the Project (LOP). As the primary implementing partner, CHAMP also reported company results as well as cumulative LOP results, both against project performance indicators. This evaluation’s data verification exercises included triangulation of reported results between original company field documentation and reports received and entered by CHAMP. For the indicators against which companies reported results, there were only minor and not significant variations between company data and CHAMP reports. At the same time the team looked for evidence of a trail of expenditures to support the results claimed by company partners and by CHAMP. The evaluation team undertook an examination of company financial documentation to support company-reported results. While the examination did not include all companies, over 50% of total GDA partners were reviewed. In most cases an expenditure trail was found to support company reported results. Companies in total reported results against 13 of the 34 PEPFAR indicators listed in the CHAMP system, representing results for activities specifically carried out by the companies. CHAMP reported results against 32 of 34 indicators, including 20 additional indicators and results for activities – primarily training and training of trainers (TOTs) managed by CHAMP/COMETS headquarters. It is telling that company GDA partners have chosen to report on the results they think most important-even after having signed agreements holding them accountable for a much broader array of targets and results. Clearly, company input had determined company effort and company results. Of the total 35 PEPFAR indicators relevant to COMETS, most partners agreed to meet targets in at least 30 of the 35 indicators. Despite the agreed targets, most companies only reported results in 10-13 indicators. It is noted that the indicators in which company partners reported greatest results are those with the most direct impact on the workforce (treatment) or which can be provided quickly and easily (sensitization and testing). Table 4 illustrates performance to indicator targets, disaggregated by GDA Partner Company. Figures in red are cumulative results below 75% of target; figures in black are cumulative results between 75% and 149% of target; and figures in green show results above 149% of target. Table 4. Company-Reported Results by PEPFAR Performance Indicators* Company 1.2 1.3 2.1 5.2 6.2 7.2 8.1 8.1B 9.2 11.2 11.3 11.4 CEC 359 359 85 49 257 Dunavant 183 78 149 155 172 409 175 35 70 72 FQML 147 119 177 181 133 93 41 320 27 36 37 Kafue 205 200 72 72 23 11 76 252 177 182 Kagem 8 9 9 8 31 12 KCM 183 153 139 139 31 46 313 22 96 89 Lumwana 5 10 98 153 3 42 691 MFA 266 38 104 104 83 11 100 72 55 57 Mopani 237 150 159 159 238 60 41 195 194 95 90 THPAZ 65 65 146 Zambeef 689 106 Zambrew 50 50 2084 ZRA 7 7 31 Zam Sugar 134 54 180 180 247 287 261 24 32 25 *See annexed list of COMETS PEPFAR Indicators for the full description of indicators. The most effective company efforts are evident. One can immediately see the indicators that were, by and large, addressed successfully (such as 9.2 – CT accessed), and those where companies had difficulty in implementing the intervention (such as 8.1 - Number of OVC served by OVC program). Overall, during the COMETS life of project, the highest performing companies were Dunavant, FQML, Kafue, KCM, Mopani, and ZamSugar. Overall, partners were relatively more successful in achieving targets for 27 Sensitization and Counseling and Testing, compared to new ART enrollment. The latter challenges include enrollment of women into ART for PMTCT (See Annex for the full set of PMTCT figures and tables, not included in the body of the report). In total, 7 companies met Sensitization AB targets, 8 met Sensitization Other targets, 10 met Counseling and Testing targets, but only two met ART New Enrollment targets (Figures 3-5). Figure 3. Sensitization: LOP Results Against Targets Figure 4. Counseling and Testing: LOP Results against Targets 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 Population Sensitization: LOP Results Against Targets Sensitization AB Sensitization Other Sensitization Target 0 10,000 20,000 30,000 40,000 50,000 60,000 Population Counseling & Testing: LOP Results against Targets CT Accessed CT Target 28 Figure 5. ART Enrollment: LOP Results against Targets It should be noted, however, that the data shown do not alone, lend themselves to a deeper interpretation of why certain companies performed better than others, or performed well or poorly against certain interventions and indicators. First, it is possible that these partners set their targets unreasonably low. More importantly, it could also suggest that there are factors that lead to success that are separate from participation in the COMETS program, but unique to this sub-group of high performers. One likely factor is program longevity. High-performing partners included Dunavant, KCM, Lumwana, MFA, Mopani, FQML, and ZamSugar. Several of these companies had programs that pre-dated the COMETS program; some of these programs were direct predecessors of the COMETS project such as BRAT and SHARe. Indeed, probing through qualitative interviews indicated that many of these partners were able to build directly on previously implemented program activities. The infrastructure and staff capacity to take on the COMETS project and carry out activities from the first day forward is an advantage over partners who may have been trying to establish their systems for the first time, or those who did not have a clear staffing structure responsible for carrying out all the planning, budgeting, and coordination required to undertake such a project and start implementing activities right away. Mining companies appeared to be more successful targeting men and agricultural companies were generally more successful in targeting women. These results make sense considering the gender composition of the workforce, but keeping in mind the explicitly stated link between the community and the workforce, the extensive spouse and dependent-oriented activities, and the gender integration components introduced by CHAMP throughout COMETS, partners theoretically were equipped to at attempt gender-balance in target achievement. Figures 6-8 present the gender-disaggregated results with respect to meeting targets, with LOP targets plotted by dividing the overall LOP targets in half to approximate a hypothetical even-split between men and women. In reality the split may have been different but no gender-specific targets were set as far as the team was able to ascertain. It is important to also consider the data from the SmartCare system in target achievement (Figures 9-10). These are additional individuals who have newly initiated ART through the Mobile Health Units (MHUs) that support rural health centers (RHCs) over the life of the COMETS project – in this case, the performance of the MHU matches with the general pattern of high and low performing partners. However, all mobile health units enrolled more women than men on ART. 0 500 1,000 1,500 2,000 2,500 3,000 Population ART Enrollment: LOP Results against Targets ART New ART New Target 29 Figure 6. Sensitization: LOP Results against Targets, Gender-Disaggregated Figure 7. Counseling & Testing: LOP Result against Targets, Gender-Disaggregated 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 Population Sensitization: LOP Results against Targets Gender-Disaggregated Sensitization AB - Female Sensitization AB - Male Sensitization Other - Female Sensitization Other - Male Sensitization Target / 2 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 Population Counseling & Testing: LOP Results against Targets Gender-Disaggregated CT Accessed - Female CT Accessed - Male CT Accessed Target / 2 30 Figure 8. ART Enrollment: LOP Results against Targets, Gender-Disaggregated Figure 9. New ART Enrollees through Mobile Health Units by District, LOP 0 200 400 600 800 1000 1200 1400 Enrollees ART Enrollment: LOP Results against Targets Gender-Disaggregated ART New - Female ART New - Male ART New Target / 2 240 281 523 84 1388 535 203 0 200 400 600 800 1000 1200 1400 1600 Gwembe (Dunavant) Lundazi (Dunavant) Solwezi (FQML, Lumwana) Mpika (Zambeef) Katete (Dunavant) Mkushi (MFA) Kafue (Kafue Consortium) New ART Enrollees through Mobile Health Units by District, LOP 240 281 523 84 1388 535 203 0 200 400 600 800 1000 1200 1400 1600 Gwembe (Dunavant) Lundazi (Dunavant) Solwezi (FQML, Lumwana) Mpika (Zambeef) Katete (Dunavant) Mkushi (MFA) Kafue (Kafue Consortium) Enrollees New ART Enrollees through Mobile Health Units by District, LOP 31 Figure 10. New ART Enrollees through Mobile Health Units by District, LOP, Gender￾Disaggregated One might also argue that trends over the life of the project are important to consider in addition to the total achievement, as improvements over time could be indicative of progress regardless of target achievement. The evaluation team considered this by looking at trends in project results over the life of the project (Figures 11-14). There appears not to be a distinctly visible or consistent pattern among all partners. Notably, for many partners, results were progressively better over a period of time, but then dropped off toward the end of the program. It is possible that partners perceive a lesser incentive to continue with programming after targets are met. While it is true that the annual average achievement often drops off after target achievement, (Table A3-1 to A3-6 in Annex VII) this is not a uniform pattern across all partners. Nevertheless, one can conceive of a scenario where a partner agrees to revise a target to be 25% higher than the original figure if the target is met within the first 2 years of the project life, and perhaps is allowed to access some small amount of supplementary funds to do so. Private sector partners appeared to be incentivized to impact their employees and surrounding communities in this way by target setting, and conversely, not incentivized properly over time to re-set targets and to achieve gender balance given the lack of dynamic and gender-oriented target setting process. It is also clear from these figures and tables that many partners sustained low levels of achievement throughout the life of the project. It is possible that low targets result in low achievement, but this would not explain why other partners meet and exceed targets by so much. Indeed, even low performing partners did not express unwillingness to engage and expressed appreciation and desire for the COMETS activities. More likely, inadequate support was provided to partners who were less experienced in implementing such programs, or more capacity-constrained in terms of staff members who knew how to coordinate activities so as to effectively meet and exceed targets. 85 123 196 37 500 230 79 155 158 327 47 888 305 124 0 100 200 300 400 500 600 700 800 900 1000 Gwembe (Dunavant) Lundazi (Dunavant) Solwezi (FQML, Lumwana) Mpika (Zambeef) Katete (Dunavant) Mkushi (MFA) Kafue (Kafue Consortium) Enrollees New ART Enrollees through Mobile Health Units by District, LOP Gender Disaggregated Male Female 32 Figure 11. Sensitization AB LOP Trends Figure 12. Sensitization Other LOP Trends 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 2009 2010 2011 2012 Population Sensitization AB FQML ZamSugar KCM Dunavant Mopani ZRA Lumwana Zambeef Kagem MFA ZNFU CEC Kitwe Kafue Consortium Zam Breweries THPAZ 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 2009 2010 2011 2012 Population Sensitization Other FQML ZamSugar KCM Dunavant Mopani ZRA Lumwana Zambeef Kagem MFA ZNFU CEC Kitwe Kafue Consortium Zam Breweries THPAZ 33 Figure 13. Counseling & Testing LOP Trends Figure 14. Treatment (ART New) Lastly, to explore the implicit mechanisms and assumptions of the project logic, the team evaluated the relationship between the achievement of results in the sensitization, CT, and treatment indicators – i.e., whether more sensitization was related to greater numbers of CT, and whether more CT appeared to be associated with greater numbers of individuals enrolling in ART (Figures 15-16). Overall, there does appear to be a positive association between levels of sensitization and CT, and then CT and ART enrollment. However, the team cannot perform a statistical analysis given the lack of a comparison scenario and feasible control variables, and these results also appear to be driven by a select few partners as in other results (see Annex VII). The team therefore cannot make definitive causal 0 5,000 10,000 15,000 20,000 25,000 2009 2010 2011 2012 Population Couseling & Testing Accessed FQML ZamSugar KCM Dunavant Mopani ZRA Lumwana Zambeef Kagem MFA ZNFU CEC Kitwe Kafue Consortium Zam Breweries THPAZ 0 100 200 300 400 500 600 2009 2010 2011 2012 Population Treatment (ART New) FQML ZamSugar KCM Dunavant Mopani ZRA Lumwana Zambeef Kagem MFA ZNFU CEC Kitwe Kafue Consortium Zam Breweries THPAZ 34 statements about these relationships, since it’s possible that some external factor could be responsible for a community’s willingness in general to engage in sensitization and testing, responsible for the increase in both. However, through triangulation with qualitative interviews, the team learned of marked shifts in stigma specifically related to counseling and testing as well as taking antiretroviral drugs both in the workplaces and communities. Health workers and HIV program coordinators working for the private sector partners offered accounts of employees being open and vocal in the workplace about their HIV positive status, setting examples for others of the importance of testing, treatment, and the possibility for regaining a healthy, productive lifestyle as well as the key aspects of “living positively.” Community peer educators also reported in the FGD that levels of condom use had been increasing, multiple concurrent partnerships decreasing, and partner participation (i.e. men accompanying their wives to antenatal care and engaging in prevention of mother-to-child transmission) or partner counseling and testing increasing in the community. Figure 15. Association between Sensitization (AB) and CT Figure 16. Association between CT and ART 2009 2010 2011 2012 2009 2010 2011 2012 2009 2010 2011 2012 0 20000 40000 60000 80000 CT Accessed: No. People 0 100000 200000 300000 400000 Sensitization (AB): No. People Male Female Total Sensitization (AB) & CT Accessed 2009 2010 2011 2009 2012 2010 2011 2012 2009 2010 2011 2012 0 500 1000 1500 2000 2500 ART New Enrollees 0 20000 40000 60000 80000 CT Accessed: No. People Male Female Total CT Accessed & ART New Enrollees 35 CONCLUSIONS  Although some constraints were identified by GDA partners, the COMETS Project was overall highly effective in implementing project activities. The presence, technical assistance and facilitation of the COMETS project served as a catalyst and provided measurable momentum to GDA workplace programs.  The GDA model was effective as a whole in terms of achieving results related to sensitization, counseling and testing, and ART enrollment, though to a lesser degree in the latter category. However, results were driven by a set of high performers and per-partner results were less consistent and did not convey a sense that COMETS participation alone contributed to the success of results achieved. More than simply having long-standing workplace programs, higher-level corporate interest and budgetary commitments were probably reinforced through participation in COMETS. RECOMMENDATIONS  Targets motivate partners to implement activities and achieve results and should be dynamic and gender-oriented in future programs.  Strategies should be devised to provide newer or more resource-constrained partners may need additional, targeted support during project implementation.  Stakeholders should recognize that many factors position partners for success, and along these lines should explore the option of a partnership readiness threshold that would serve to “qualify” partners for the program. LESSONS LEARNED  The GDA model leverages the ability of private partners to reach populations and the ability of other partners to provide the technical, financial, and material resources that need to be distributed. The GRZ should continue to explore the application of the GDA model for HIV programming, with a view to expanding into other private sectors.  GDA partners who are willing and able to participate should be encouraged to compete for government resources. For those wishing to apply the GDA model to HIV programs, may wish to invest effort into developing an index of factors that could be used as a readiness threshold. This may also make some partners eligible to help newer partners, or other contractors and smaller businesses, develop their programs as appropriate and feasible. USAID should consider only bringing into the GDA companies that present a specified threshold for readiness to participate, and be prepared to work with fewer companies with demonstrated interest, understanding and commitment. A few robust cases are better than more, mediocre performers, and is also going to minimize the risk of adverse effects on communities who start to receive services and then see them discontinued.  Since many factors impact behavior change among beneficiary populations, stakeholders involved in the formulation of a model for developing PPPs should consider strategies to measure such competing factors throughout the life of the project with a view to increasing the ability of an evaluation to rule out alternative explanations for observed results. Indicators should also be closely formulated to overall and specific project goals. 36 3: REDUCING THE WORKPLACE IMPACT OF HIV/AIDS Evaluation Question 3: What has been the project’s contribution to reducing the impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? FINDINGS Partners, both high and low performing, clearly articulated the importance of HIV workplace programs, including COMETS, in reducing the workforce burden of HIV/AIDS. Many of these companies cited exactly the type of quantitative indicators the evaluation team sought to obtain – including absenteeism, productivity, and retention, as well as reductions in medical claims and funeral costs. While quantitative data was harder to obtain for reasons discussed earlier, the team was able to assess Zambia Sugar productivity data. There is no apparent and consistent trend across productivity areas except for a clear decline in turnover during the life of the project (Figure 17). Figure 17. Employee Retention and Leave for Zambia Sugar, LOP With respect to medical expenditures (Figure 18), the team emphasizes several caveats in linking this kind of information with COMETS achievement. First, medical expenses and employee leave are taken completely out of context without being able to connect these numbers with any epidemiological or medical record data for the workforce. Given the companies’ policy to not discriminate based on HIV or any other illness, separating which leaves are for which causes of illness is not possible with company data. 223 1984 2378 1943.5 741.5 661.5 659.5 520 19812 21714 22134 18740 9352 11187 12101 8788 2.1 1.8 1.6 1.2 0 0.5 1 1.5 2 2.5 0 5000 10000 15000 20000 25000 2009 2010 2011 2012 to date Employee Retention and Leave over LOP (Zambia Sugar) Skilled Paid Sick Leave (days) Unskilled Paid Sick Leave (days) Skilled Paid Non-Sick Leave (days) Unskilled Paid Non-Sick Leave (days) Turnover % 37 Figure 18. Medical Expenses for Zambia Sugar, LOP The team acknowledges that CHAMP is currently in the process of repeating a very detailed effort to conduct a Cost-Benefit Analysis that includes granular data on these indicators that will be compared to a baseline CBA conducted before the start of the COMETS program. At the time of the evaluation, this information was not yet available. This may provide additional insight into this evaluation question. It is also worthwhile to note that this evaluation question could encompass an entire, comprehensive analysis by itself, that could include longitudinal data from the partner perspective, the community perspective, and other supplementary data such as district-level epidemiological data and other regional economic data, all of which could be combined and used to put productivity and workforce data into perspective both in terms of context as well as over a period of time that is reasonable in terms of the cause and effect relationships that are hypothesized. Qualitative evidence for the impacts of the HIV workplace programs on these indicators, however, is abundant. All of the private sector partners interviewed, as well as other key informants, consistently report these exact indicators as original impetus for instituting HIV workplace programs, and for participating in public-private partnerships. In terms of the effect of increasing CT access and ART enrollment, Human Resources professionals that were interviewed provided evidence that absenteeism, most notably chronic and prolonged absenteeism, had been reduced over the last few years. There is no reason to doubt whether advances in ART delivery, technology, and adherence has been a factor in this reported trend, but the exact contribution of the COMETS program is difficult to isolate. This includes participation for larger partners in earlier iterations of the GDA model as well as this COMETS project. It is worthy to note, however, differences between the accounts of mining and agribusiness partners. In the mining sector, these indicators are visible to the company management as well as other employees in a way that is much more apparent than in the agricultural sector. Dunavant, for example, noted that part of the difficulty in producing the data requested was that by and large their workforce consists of farmers. Of course, this does not include company staff on payroll and headquarters and international staff, but those with the most influence over production are the contracted farmers dispersed around the country. Key informants from Dunavant noted that they clearly do not track when the farmers are out in the field, but rather measure productivity downstream, in terms of yield produced. 0 500,000,000 1,000,000,000 1,500,000,000 2,000,000,000 2,500,000,000 2009 2010 2011 2012 to date Zambia Kwacha Medical Expenses over LOP (Zambia Sugar) Referrals (K) Treatment (K) 38 The evaluation team also acknowledged that CHAMP conducted a Knowledge, Attitudes, and Practices survey in the surrounding community. The KAP tool that the evaluation team had access to was conducted in 2010. Unless such a survey is conducted in 2008, and 2012, such measurements mid￾program cannot supplement a rigorous evaluation. The value added of being able to conduct an analysis looking at all the partner data side by side would include being able to make more general statements about trends across all GDA partners, but also the ability to compare these indicators across partners who had been participants in the predecessors of the COMETS programs. It may be the case that the effects on absenteeism or spending on health are lagged after a certain amount of time. All of this being said, it is also important to acknowledge the established link between antiretroviral treatment and recoveries of health and the potential for productive work and activity. The relationship between effective, adherent treatment regimen and gains in health status are well-documented in the literature and continue to be studied with the same substantial amount of attention as a proper sub￾discipline within medicine and public health. Therefore, it is reasonable to look at the achievements that occurred within the life of the COMETS project in terms of counseling and testing and antiretroviral therapy initiation, and conjecture that quality of life will have been increased discernibly over the period between 2008 and 2012 given the large amount of individuals put on treatment. This kind of conjecture is highly dependent on some strong assumptions, of course, including that these individuals were put on treatment early enough to recover to a productive health state, that they continue to have access to treatment, that they adhere to drugs, and otherwise maintain healthy behaviors key to positive living such as good nutrition. CONCLUSIONS  The workplace burden of HIV was reduced through COMETS as a continuation of other programs that predated it. This conclusion is overwhelmingly supported by qualitative evidence. Private sector partners consistently report these exact indicators, nonetheless, as original impetus for instituting HIV workplace programs, and for participating in public-private partnerships – both in earlier iterations of the GDA model among applicable partners and in the COMETS project. Quantitative data to support this claim was not available for more than one partner during this evaluation. RECOMMENDATIONS  Company emphasis on providing Counseling and Testing and referrals or direct treatment services to employees should be continued. Efforts to partner with the public sector or other stakeholders that can assist companies to continue these services is paramount considering the strong link documented in the literature between medical treatment for HIV and the increase in productive lifespan. Given that this link depends on adherence and other health lifestyle factors, companies should also ensure to continue providing any other psychosocial services in the workplace that contribute to employee wellness especially in the context of HIV. LESSONS LEARNED  Since company data and medical records are very sensitive, evaluators are not likely to be successful in collecting this data strictly ex-post. Strategies to collect this data confidentially (as with other categories of data mentioned earlier) can be integrated into future GDA program monitoring and evaluation structures.  Those formulating methods for evaluation GDA/PPP models should consider supplementing private company data with systematic and repeated community-based quantitative surveys in the surrounding areas. Validated questionnaires and surveys already exist for workplaces and households, and can provide valuable information regarding illness episodes, sick leave, productivity, employment, stigma/discrimination, health care costs, workplace benefits, and the like. Evaluations 39 planned earlier in the life-cycle of the project would hold the most promise in terms of ensuring that all possible avenues for quality data collection are attempted. 4: EFFECTIVENESS OF WORKING WITH LOCAL IMPLEMENTING ORGANIZATIONS AND PRIVATE SECTOR PARTNERS Evaluation Question 4: To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost-efficiency, and monitoring and evaluation? FINDINGS In order to assess whether the approach taken by USAID to work with local implementing partners was effective, and to evaluate its strengths and weaknesses, the evaluation team used the following components of successful GDA model approach as a benchmark reference:  Identify partnership goal(s) that address the needs or core function of each partner  Identify each partner’s unique strengths and the specific contribution each partner can make toward achievement of the partnership ability to achieve goal  Identify each partner’s self-interest  Align the partnership to the strongest partner’s self-interest first; then align partnership goals to the remaining partners’ interest  Identify and create a PPP strategy that meets the needs/core interest of each partner  Create PPP goals and targets that are both short and long term, are measureable, and hold each partner accountable for agreed upon levels of performance  Measure performance, create strategies to remediate when needed, and report performance to all partners  Long-term continuance of the PPP is anticipated if short term performance measurement shows the continued value of each partner. Forming and Maintaining Relationships The GDA model of public-private partnership was found to demonstrate substantial ability to achieve broad project goals, despite structural challenges. There is substantial qualitative evidence that COMETS harnessed robust corporate involvement in workplace programming and influenced and coordinated HIV programming. Specifically, partners reported increased ability to achieve company goals more efficiently as a result of direct cooperation and involvement of USAID. COMETS was found to be a critically important coordination vehicle for intra-company cooperation. For example, mining partners visited noted that accreditation of health facilities and the acquisition of supplies from the government medical stores would have been a much lengthier and more difficult process had it not been for the combined leverage of the partnership and the centralized coordination of the implementing partner, CHAMP. In addition, COMETS provides a model for new private sector entrants into partnerships for workplace health programming. The team was provided several examples of smaller companies seeking information on joining workplace programming; larger private sector partners additionally expressed the desire to bring organizations, contractors and/or other close partners into such a partnership. In addition, companies expressed that partnerships would allow for a more equitable sharing of financial responsibility. For instance, many companies run health facilities that provide services to anyone in the 40 community, regardless of whether they are employed with the company. By bringing these organizations on board within an partnership, companies comprising the existing network may be able to reduce their level of financial commitment somewhat by asking other companies to contribute toward the same ongoing costs, thereby spreading responsibility, ownership, risk, empowerment, and information among all members of the partnership. COMETS also provided a useful model for building new programs upon the successes of predecessors. It was clear that partners who had established HIV or other health programs were prepared to implement immediately upon program initiation. Those partners who participated in the earlier iterations of the partnership that began from 2004, or had internal programs that dated even farther in the past, possessed a pre-existing understanding of their goals with respect to the use of COMETS resources – not simply in terms of financial contribution, which is small compared to company input, but mostly in terms of how to leverage their participation in the partnership. This means accessing national medical stores, forming linkages with health facilities in the community, implementing mobile health units, and accessing a network of health educators in the community. Grants Management CHAMP/COMETS developed memoranda of understanding (MOU) with 14 GDA partners, each of whom committed financial and/or in-kind contributions and submitted a list of equipment that they needed to ensure the sustainability of the HIV/AIDS program. CHAMP feels that the partnerships have been a success overall. Despite the fact that some of partners did not meet their targeted financial contributions, the total financial contributions achieved by all partners exceeded total commitments. One partner, Zambeef, withdrew from the partnership; thus, its financial information is unavailable. COMETS arranged for and supported the provision of MHUs that functioned as partner communities. CHAMP seconded staff to MHUs, while the partners provided unit maintenance and MOH provided medicines. This collaborative approach precluded partners from needing to hire health staff directly, allowing partners to harness a key resource efficiency. CHAMP staff offered technical support assistance to enable the partners to implement activities in the workplace and at sites. The activities include training, sensitization, counseling and testing. CHAMP also provided partners with M&E officers to help with data collection and consolidation, as well as the preparation of monthly reports. According to an interview with a GRZ representative, in practical terms, GDAs and COMETS contributed to the national health plans primarily by participating in district level health planning and management meetings, and providing financial and coordinative support for the operation of MHUs, thereby expanding access of HIV/AIDS services to rural areas, as called for in national health plans. The representative expressed deep appreciation on behalf of the GRZ for the assistance provided through COMETS and the GDAs. However, the GRZ felt CHAMP/COMETS had not fully anticipated GRZ/MOH policy and budgetary limitations regarding support and maintenance of mobile units. As the representative explained, it is GRZ/MOH policy, because of severely limited resources, to place lesser priority on mobile units than on building permanent infrastructure, and thus it proved difficult to find GRZ/MOH resources to transfer ownership and absorb the recurrent running costs of the mobile units. This led to difficulties in CHAMP/COMETS arranging handover to the GRZ – a situation that had not yet been resolved by the end of the COMETS project. With regard to the implementation of COMETS, CHAMP reported two major challenges, both related to personnel and the limitations of resources available to CHAMP as a local implementing partner. First, job insecurity for employees caused staffing instability, especially towards the end of the COMETS project, when many people left to join other organizations offering long-term employment prospects. Attracted by higher salaries, many CHAMP staff also left to pursue employment with the government, demonstrating the poor financial incentives available to CHAMP to retain staff. 41 Second, as discussed with the evaluation team, CHAMP management felt that it entered into its Cooperative Agreement with USAID with a disadvantage as a local implementing organization. This disadvantage concerned the fact that CHAMP did not originally have the necessary management system infrastructure in place to efficiently and effectively cope with the additional management responsibilities required of a COMETS grants manager. CHAMP/COMETS indicated that project management systems could have been more efficient and effective if an adequate cadre of management specialists had been hired initially and retained. CHAMP claimed that its existing management systems were “stretched”, and would have liked to hire for additional capacity. However, CHAMP was limited in expanding its core management support personnel because it did not receive a Negotiated Indirect Cost Rate Agreement (NICRA). Procurement Procurement for GDA partners was undertaken with a view to ensure the sustainability of workplace HIV/ AIDS programs after the termination of COMETS, as well as to build both human and technical capacity. Equipment and human capacity building resources were awarded through a procurement process, which met Zambian and international audit standards. As part of their MOU agreements, partners were asked to submit a list of equipment necessary to ensure the sustainability of the HIV/AIDS program once the COMETs project came to a close. Partners were additionally asked to commit to achieving specific program performance targets, as outlined under the GDA action plan. Once MOUs were signed, CHAMP and the partners carried out regular monitoring of their performance against each activity, as well as against the established financial contribution. Thus, the decision to procure equipment for the partners rested on the whether the partner had met the program performance targets outlined in the GDA action plan, as well as meeting or exceeding the financial contribution to which they committed. Partners failing to meet the above criteria were asked to ramp up their performance during the COMETS 4th year no-cost extension period, or they would not be awarded grants procurement. GDA partners also benefited from the procurement contract that CHAMP had with Medical Stores Limited (supported by the MOU with MOH) for the provision of medical equipment and supplies ranging from male and female condoms to HIV test kits. According to CHAMP, a major success story of grant procurement was the provision of a CD4 count machine to Mopani Copper Mines, which is accessible to the district and neighboring towns of the Copperbelt. In addition, the upgrade of the Mobile Health Units (MHUs) from standard to all-weather vehicles ensured that they were able to penetrate into otherwise inaccessible parts of Zambia and provide ART and general health checks to rural communities. However, substantial challenges with regard to procurement were reported by partners, who expressed concerns about delays in acquiring equipment due to CHAMP’s high staff turnover rate. The evaluation team visited a health facility near the main headquarters in Kitwe, in which a PCR machine used for counting viral load sat, non￾operationally, because of the lack of agreement on how to procure reagents for the machine. The COMETS project, therefore, had provided support for the purchase of this machinery with the understanding that the partners would then take responsibility for purchasing reagents as needed for the machinery. Mopani expressed that procuring support for recurring program costs, such as reagents, was a main factor driving their desire to partner with other organizations. Communication A significant challenge in working directly with local implementing organizations, as reported by most partners, was persistent lack of communication between partners and CHAMP representatives. Partners expressed frustration regarding the lack of follow-up on training opportunities, as well as a lack of responsiveness from CHAMP when inquired about future opportunities. Requests for information and 42 resources on behalf of partners to CHAMP were often not addressed, and in some cases, partners’ unfamiliarity with CHAMP’s general structure prohibited them from making contact at all. Monitoring and Evaluation COMETS was found to have identified partner core business needs and competencies, as well as understood each partner’s unique interests and contributions. However, COMETS did not align programmer goals with those of the private sector; COMETS performance targets were not directly tied to workplace productivity or health indicators. Rather, performance targets were aligned with a set of standard PEPFAR indicators, and most results reported by company partners were for activities directly related to company-specific interests. A majority of PEPFAR indicators did not show company reported activity or results. The discrepancy between program goals and company goals, for example, can be seen with respect to OVC care. CHAMP reports on an aggregate number of OVCs served over the life of the project, but it was not possible to verify this against partner records. The project’s information systems were entirely focused around reporting and accounting for PEPFAR indicators. As commented to the evaluation team by more than one GDA partner, PEPFAR reporting as a basis for M&E (“numbers counting”, as it was called) did not provide information for strategic program management and systemic change. A primary example of this gap was the fact that no provision was made early in the project to set a baseline and routinely gather data to support a cost-benefit analysis to measure improvements in the productivity of workers as a result of the COMETS-led interventions – a critical business model issue. The evaluation team acknowledges CHAMP’s effort to conduct a cost￾benefit analysis before and after the program, which will provide a valuable source of data related to productivity indicators and will allow for critical before-after comparisons. However, in the absence of data that can be used to eliminate alternate explanations for any changes observed, before- and after￾data is not sufficient to attribute changes solely to the COMETS program. While visibly well-structured, the CHAMP/COMETS M&E system was not found to be operated efficiently. All program data was captured at the head office by a team of M&E Data Clerks, and on-site for activities undertaken by field teams. CHAMP employed a system of routine site supportive supervisory visits and conducted capacity building for its staff and partners to ensure that COMETS monitoring was carried out efficiently and effectively. Substantial monitoring and evaluation challenges were faced over the life of the project. In collaboration with its partners’ focal point person (Company HIV Coordinator), CHAMP introduced the project Technical Support Assistant and the M&E Data Clerk at each GDA partner site. Partners reported that this staff member was of the utmost utility during the life of the project, removing the burden of the USAID- and PEPFAR-format reporting from company payroll staff. However, staff lamented the poor communication regarding the removal of this seconded individual at the end of the project. Trainings were planned and undertaken in order to build capacity in￾house at the corporate partner offices, but did not necessarily address the fact that such individuals may already be pressed for time in their daily responsibilities. Monitoring and evaluation procedures were not responsive real-time to track progress of partners, and the overall systems were not necessarily aligned with partner priorities as they did not find it necessary to hire M&E professionals after the COMETS project was finished. Peer health educators from Mazabuka, working with Zambia Sugar, expressed that CHAMP staff would arrive when the need for data collection arose, but would not otherwise regularly communicate with the peer educators about their work, challenges faced, and other needs. For example, bicycles are a necessity in this area to traverse communities widely dispersed across large geographic areas. When the need for more bicycles or bicycle repairs arose, peer educators expressed that they received very little, if any, response from CHAMP. Peer educators were then expected to sensitize the community and return with the raw data for routine monitoring and evaluation checks. 43 CONCLUSIONS  Overall, USAID’s approach of working directly with local implementing partners has been effective in terms of promoting the achievement of workforce HIV programming goals and increasing stakeholder buy-in and accountability to overarching program objectives.  Management of the COMETS project by CHAMP was overall successful, with respect to fostering and maintaining relationships with private sector partners. Chronically poor communication, however, presents a significant challenge to cooperation within partnerships.  Other significant challenges were faced in the areas of staff retention, monitoring and evaluation, and procurement procedures. Procurement was not nearly as efficient as private partners would have been satisfied with, which influences future opportunities to continue these partnerships. Monitoring and evaluation professionals were not valued highly enough to be made a permanent part of private sector partner staff post-COMETS.  USAID’s cooperative approach was particularly strong in supporting GRZ national health plans by participating in district health planning and management meetings, and by providing financial and coordinative support for running of mobile health units, thereby expanding access to HIV/AIDS services to rural areas. However, due to lack of GRZ capacity, CHAMP/COMETS experienced difficulty transferring ownership of MHUs to the GRZ. RECOMMENDATIONS  Partners with existing HIV workplace programs or health departments were ready to implement programs early, and were most successful in terms of managing the COMETS project and mitigating the management challenges. Additional, intensive, and up-front support to new partners with respect to managing such projects or getting health departments off the ground would most likely help private partners prepare for such projects, while alleviating some of the management burden from the implementing partner.  Companies should be willing to hire an additional monitoring and evaluation professional if they desire to continue tracking progress on key indicators after COMETS program termination, or otherwise explicitly allocate sufficient staff time to carry out these responsibilities.  More attention should be placed on how to appropriately incentivize all levels of program staff to routinely record and collect data, with built-in quality assurance processes in place from program initiation through close-out. LESSONS LEARNED  Monitoring and evaluation is a process that cannot exist in a vacuum without any connection to the context in which such data is generated and collected. Methods of incentivizing data collection should conform to the needs of program staff (e.g. bicycles), and attempt to account for unanticipated externalities (e.g. the need for bicycle repair). The ability to track program inputs and outputs in real-time is of critical importance to forming the basis of future programming decisions; therefore, providing appropriate human and technical resources to undertake routine M&E should be prioritized.  In instances where a local implementing partner is providing inputs (such as MHUs) that imply the host government will eventually absorb recurrent costs of said input following project completion, every effort should be made early in the project to ensure that an MOU with the government includes the requirement for government absorption of the item into its budget prior to the end of the project. 44 5: ADDED VALUE OF USG PARTNERSHIP WITH GDA PARTNERS Evaluation Question 5: Besides the USG funding contribution, what has been the added value, if any, of the USG partnership with GDA partners? (A) What has been the added value to private companies, GRZ, and local communities? (B) What additional benefits have been realized in engaging private companies that already have corporate social responsibility? (C) What multiplier effects have the USG-GDA partnerships had within and outside of HIV/AIDS? FINDINGS (A) Added Value to Private Companies, GRZ, and Local Communities The COMETS model added the technical competence of CHAMP and its community-based HIV/AIDS interventions to the existing USAID experience of working with the private sector to achieve results. By partnering with CHAMP to implement COMETS, USAID attempted to build new HIV/AIDS workplace programming on the shoulders of prior effective workplace programing with the private sector and an effective community-based HIV/AIDS education organization. Overall, local communities were found to have benefited substantially from COMETS as a result of expanded private sector programming. Specifically, working directly in the community was cited as a means of reducing stigma, tracking patients more efficiently, and sharing program ownership. One company was provided the resources to expand ART to farmers in areas otherwise considered too difficult to reach. Another company cited the ability to serve as a repository for other organizations’ supplies as a significant value add to the community, aiding in the distribution of condoms throughout remote areas. Companies were also provided the flexibility to achieve program targets by creating customized, company-specific activities, which allowed some companies to expand programming further into surrounding communities. Increased community outreach was found to stimulate interest in volunteerism, though in the absence of performance incentives, volunteer retention presented a challenge. Several partners expressed appreciation at the opportunity that participation in the COMETS program provided, with respect to allowing them to now think about expanding their programs more holistically to include other health areas, including malaria and maternal and child health most prominently. Some partners are already engaged in such activities, and have found positive externalities of the COMETS program to be applicable, such as lessons learned in terms of monitoring and evaluation and linkages with local health facilities. One partner, Zambian Breweries, expanded its programs to specifically benefit women in surrounding communities by subcontracting a local women’s empowerment NGO for work in three company locations. (B) Additional Benefits Realized by Engaging Partners with Corporate Social Responsibility COMETS was seen by most private sector entities as a valuable addition of HIV/AIDS education and services that were complementary to existing workforce programming. The addition of USAID funding was welcomed by participating company partners. COMETS captured the community-based programming strength of CHAMP to broaden the reach of workforce health programming to the surrounding communities. Corporate partners were able to reach out to their surrounding communities in a way that may not have been otherwise supported by the company’s budget, regardless of the availability of funds. Some companies, for example, took in the COMETS funding and managed and coordinated the activities from 45 within their Corporate Social Responsibility arms or departments, even if the activities were seen as essential to maintaining company operations (by maintaining the health of their workforce). Partners were found to have capitalized on having the social license to operate in the community. HIV programs were often initiated under a Corporate Social Responsibility umbrella at corporate partners, rather than being integrated into company operations through Human Resources departments. Partners reported that it is sometimes easier to justify such activities and protect company investment in such programs by keeping it within the realm of CSR, as doing so provided companies more flexibility in using such program budgets without working within the main company structure. However, through detailed discussions with partners on this topic, the team understood that regardless of where the program was officially housed, most management staff viewed such programs as part of company operations at the current time. HIV programs, in other words, are considered essential within the workplace because of the benefits for the current workforce, while community activities were seen as essential because of conferring benefits to potential future workforce, while also maintaining the health of the families of employees, who might otherwise need time off to care for the sick; in addition, staying in favor with the community provides them with a social license to operate in terms of their core business, as well as a way to stay connected to their surrounding environment. Through these and other examples, it becomes clear that companies have moved to embrace HIV programs in a more entrenched way within the workplace over the life of the project. (C) Multiplier effects of USG-GDA partnerships within and outside of HIV/AIDS Catalytic benefits, or the unexpected value derived from activity that results from a major intervention such as COMETS, is not normally subject to a quantitative analysis. However, there were substantial qualitative findings from nearly all of our interviews that pointed to catalytic benefits. In particular, several mining ventures combined malaria prevention and treatment with HIV workplace efforts as complementary “wrap-around” programming. The quarterly forums for company-to-company sharing of workplace health efforts had a benefit for participating companies; substantial credibility for large-scale workplace health interventions was gained by the publicity and presence of Zambia’s forefront industrial partners in COMETS. At the country level, the GRZ recognized that a successful national response to HIV/AIDS required the support and engagement of stakeholders across all sectors of the economy. The private sector (including CHAMP) was singled out as having a major role to play in addressing the HIV/AIDS pandemic, with its comparative advantage of being able to develop, finance, and implement HIV/AIDS interventions in the workplace, was noted.21 Table 5 indicates input by company compared to the company’s reported number of high performing indicators (over 90% of target). Due to the lack of timely financial reporting by half of the COMETS GDA partners, it is difficult to make conclusive assumptions about the relationship between total input and results for the GDA/PPP model as a whole. However, it can be seen from the data that five of the six companies with the largest total input also showed the highest results. Five of six companies with the lowest percentage of USG input to total input (i.e. the largest percentage of company input) also had the highest results by indicator. Not surprisingly, perhaps, the three GDA partners with the highest percentage of USG input also had the lowest percentage of high performing results against indicators. COMETS financial management highlighted that while First Quantum Mining (FQML) had not completed submission of Final Year 3 financial reports, their contribution for Year 3, and in total, would substantially exceed MOU company contribution agreements. 21 The National HIV/AIDS/STI/TB Council, www.nac.org.zm/index.php/about-us/funding-mechanisms, 2009 46 Table 5. Comparison of Financial Inputs to Results across GDA Partners Pre-COMETS GDA Partner Company Agreed company input Actual company input Agreed USG input Total USG % of total # indicators over 90%/total GDA partners who have submitted final financial reconciliation Yes CEC 176,000 998,997 51,019 1,226,016 4% 3/12 Yes Dunavant 1,341,240 1,469,060 450,000 3,260,300 14% 6/12 No Kafue 100,000 198,416 60,000 358,416 17% 5/12 Yes Mopani 3,411,749* 5,359,496 170,000 5,529,496 3% 8/12 Yes KCM 3,001,735 n/a 200,000 3,201,735 6% 7/12 No ZRA 100,000 546,360 50,000 696,360 7% 0/12 Yes ZamSugar 924,693 1,429,529 115,299 2,469,521 5% 6/12 GDA partners not yet reporting Year 3 company contribution Yes FQML 1,184,003 n/a 263,000 1,447,003 18% 7/12 Yes Kagem 300,000 n/a 81,000 381,000 21% 0/12 No Lumwana 400,000 n/a 105,383 505,383 21% 3/12 Yes? MFA 707,172 n/a 4/12 No THPAZ 0 n/a 125,000 125,000 100% 1/12 No ZRA 100,000 546,360 50,000 696,360 7% 0/12 No ZamBrew (Financial data not provided) 1/12 *- Increased during LOP from MOU commitment of USD 1,705,848 n/a - No financial reconciliation available From the above table, there does not appear to be a positive correlation between degree of USG input and success, but there is some indication that larger total input leads to greater overall success. There appears to be a direct correlation between high performance and the percent of company input into workplace programming. There is also some indication of an inverse correlation between high USG input and performance. Two of the four lowest performing companies received the highest percent of USG input: Kagem Mining and THPAZ, the Zambian Association of Traditional Healers. The fact that partner contributions generally dwarf USG financial inputs is a positive indicator of the depth of private sector commitment to the goals of workplace HIV programming, especially if the commitment is sustained over time. However, the predominance of company contributions in work place programming means that the attribution of USG financial is difficult. The COMETS GDA structure and the structure of the initial MOU between COMETS and GDA partners did not require disaggregated results by company/USG; therefore, it is not possible to directly attribute results to financial contributors. Results are GDA partnership aggregates. Table 3 also shows companies that participated in GDA activities related to HIV/AIDS prior to COMETS. In company interviews, many reports indicated that prior participation enabled strong best￾practice information sharing between companies. The quarterly and annual GDA partner meetings were an important regular forum for health care strategies and cross-cutting company-to-company advice on a range of HIV/AIDS policies and interventions. Longevity of workplace programming, combined with a structured mechanism for best-practice sharing appears to enhance performance. Another major innovation of COMETS was the establishment of Community Learning Centers, which provide a central resource for members of the community through support desks, which include alcohol- and drug-harm reduction and a victim support unit for human trafficking, among many health and social services. 47 CONCLUSIONS  The most profound value of the USG partnership with GDAs, as implemented in COMETS, was the ability to build upon prior private sector engagement with their workforce for health interventions in HIV AIDS planning. The COMETS model added the technical competence of CHAMP and its community-based HIV AIDS interventions to the existing USAID experience of working with the private sector to achieve results.  Another value addition of the USG/GDA partnership model included partnership with some of the largest industries in Zambia; successful partnering between GRZ and private sector entities increases the credibility of PPP interventions as a model for catalyzing private sector engagement, community outreach and HIV/AIDS program expansion.  The intent by GDA partners to attract newer, small enterprises and share knowledge between established and newer partners adds further value to the PPP model, enabling GDAs to fuel the development of technical capacity and small enterprise growth.  The USG/GDA partnership has added value to local communities as a result of providing private companies the means to scale up community programming, specifically into areas such as malaria, maternal and child health, and women’s empowerment.  COMETS was successful in creating synergies with religious, sports, social and labor organizations for community HIV awareness programming.  Because of the size of private sector commitment, often as part of even larger workforce health care delivery, company HIV/AIDS programming (and results) was often specific and focused, emphasizing short-term interventions that directly contributed to a healthy workforce.  Degree of USG input does not necessarily translate steer company success, but there is some indication that larger total USG financial input leads to greater overall success.  Company success is a key determinant of financial commitment to corporate social responsibility, as demonstrated by a positive correlation between high company performance and the percent of company input into workplace programming. RECOMMENDATIONS  The USG/GDA model should be sustained in order to maximize private sector engagement and improve community programming, while focusing greater attention on the priorities and specific strengths of each set of private sector partners in context.  Future USG/GDA models should actively promote the expansion of workforce programming into priority areas of the community, rather than focus on short-term interventions directly related to workforce health. Since GDAs already demonstrate they are willing and able to expand programming into local communities, the addition of goal-related incentives may provide further encouragement. LESSONS LEARNED  Providing financial support for HIV programs can add additional value to community health initiatives beyond the realm of HIV. The USG/GDA partnership shows that private sector companies, given the capacity to improve workplace HIV programming, are capable of extending financial benefits to the community to support health initiatives in other areas.  The design of future HIV interventions should take into consideration the demonstrated success of the USG/GDA partnership model as a means of promoting private sector engagement in socially responsible activities.  Substantial financial input can influence the degree to which private sector partners are able to reach their programming objectives. Also, higher performing partners will invest more in workplace HIV programming, so there may be value to incentivizing high company performance. Annex . 1 ANNEXES Annex . 2 ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK C.1 BACKGROUND HIV remains one of the biggest challenges facing the development and health sectors in Zambia. The 2007 Zambia Demographic and Health Survey estimated that the HIV prevalence rate among adults is 14.3%, approximately 1.1 million people in the age group 15 to 49. To combat the global HIV/AIDS epidemic, the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) was established in 2003 and Zambia was chosen as a PEPFAR focus country in 2004. PEPFAR in Zambia promotes a comprehensive set of prevention, care and treatment activities throughout the country to support the multi-sector response of Zambia’s Ministry of Health (MOH) and National HIV/AIDS/STI/TB Council (NAC). Since HIV/AIDS touches all aspects of Zambian society, USAID/Zambia’s response to the epidemic extends beyond a focus solely on health into other arenas, including the private sector. Between 2004 and 2008, USAID/Zambia established partnerships with a number of private sector companies, including numerous workplace programs and HIV/AIDS Global Development Alliances (GDAs) and public-private partnerships (PPPs) within the mining, agribusiness and tourism sectors. These partnerships leveraged millions of dollars of resources, expanded innovative clinical, mobile and community HIV/AIDS prevention, care and treatment services through the private sector, and contributed significantly to achieving PEPFAR targets. Encouraged by these positive results, USAID/Zambia decided to continue and expand the GDA/PPP model. C.2 PROJECT DESCRIPTION The Community Empowerment Through Self-Reliance project (COMETS) was awarded as a cooperative agreement through the New Partners Initiative (NPI) on September 19, 2008 with a completion date of September 30, 2012. The life-of-project amount is $9,000,000. COMETS was awarded to the Comprehensive HIV/AIDS Management Program (CHAMP), a private, non-profit indigenous organization established in 2002 to combat the HIV and AIDS epidemic in Zambia. CHAMP was a sub-grantee under USAID/Zambia’s HIV/AIDS multi-sectoral project, “Support to the HIV/AIDS Response in Zambia (SHARe)”, from 2004-2008, and has since expanded its services through global development alliances GDAs with mining, agribusiness and tourism companies in Zambia within the COMETS project. Building on multi-sectoral partnerships begun during earlier programs, COMETS was designed to demonstrate the pivotal role that local organizations and PPPs can play in attaining Zambia’s goals of reducing both HIV/STD transmission and the socio-economic impact of HIV and AIDS, while increasing access to quality HIV/AIDS and STI prevention, care and treatment interventions. COMETS has two overarching objectives:  To support the private sector GDA partners in strengthening their HIV workplace programs and expanding programs in the local surrounding communities22.  To reduce the impact of HIV/AIDS in the agribusiness and the mining sectors by addressing issues of increasing productivity, reducing absenteeism, retention of skilled employees and labor and increased business output. 22 Under COMETS, the local community is defined as the GDA partner community population living within the catchment area of a public rural health center (RHC) as identified by the Ministry of Health’s District Health Office. The size of such communities feeding into a RHC typically ranges between 5,000 and 15,000 individuals. Annex . 3 COMETS was designed under, and contributes to, the USAID/Zambia 2004-2010 strategic objective of Reduced Impact of HIV/AIDS through a Multi-sector Response, and associated intermediate results (see Section J.1). In addition, COMETS contributes directly to Zambia’s annual PEPFAR prevention, care, treatment and systems strengthening targets. For the full project description, please refer to Annex II. C.3 PURPOSE AND USE OF THE EVALUATION The purpose of this evaluation is to assess program effectiveness in order to improve future programming utilizing public-private sector partnerships. The following list represents the intended use of the evaluation for various stakeholders: 1. USAID/Zambia: To identify promising practices and areas for improvement and to inform the design of potential future HIV interventions under USAID/Zambia’s Country Development Cooperation Strategy. 2. USAID/Global Health Bureau, PEPFAR, and other partners in health: to demonstrate effectiveness of the partnership approach with the private sector and local organizations in HIV programs. 3. USAID/Policy Planning and Learning Bureau: to suggest a model for evaluating public-private partnerships. 4. Government of the Republic of Zambia (GRZ): to demonstrate the effectiveness of the public￾private partnership model as a potential methodology for a sustainable HIV response by the government. 5. CHAMP/COMETS and GDA Partners: to inform the strengthening of its partnerships and programs. The data and findings of the evaluation will be the sole property of USAID. USAID/Zambia will disseminate the report widely, including in the Development Experience Clearinghouse (DEC), with relevant stakeholders and project beneficiaries and may use the evaluation in the design of future projects and publications. C.4 EVALUATION QUESTIONS The key evaluation questions are: 1. To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? 2. How effective has the Global Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? 3. What has been the project’s contribution to reducing impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? 4. To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost-efficiency, and monitoring and evaluation? 5. Besides the USG funding contribution, what has been the additive value, if any, of the USG partnership with GDA partners? a. What has been the added value to private companies, GRZ, and local communities? b. What additional benefits have been realized in engaging Private companies that already have corporate social responsibility? c. What multiplier effects have the USG-GDA partnerships had within and outside of HIV/AIDS? C.5 EVALUATION DESIGN AND METHODOLOGY Annex . 4 C.5.1 Evaluation Design Given the normative and descriptive nature of the evaluation questions above and the fact that a counterfactual (control or comparison group) was not established at the beginning of the project, USAID anticipates use of a non-experimental design for this evaluation. Baseline and other data are available on some of the project’s performance indicators. Where data allows, before-after comparisons will be made and in cases where no baseline data exists, the evaluation team will explore the feasibility of reconstructing baseline information. Offerors are required to propose a detailed evaluation design and methodology as part of their Milestone 1 deliverable. C.5.2 Data Collection Methodology The evaluation will use a combination of quantitative and qualitative methods to answer the evaluation questions above. Illustrative data collection methods may include:  Literature review and content analysis: Before embarking on the evaluation, the evaluation team will review all the relevant documents and extract any data that may be useful to the evaluation. Particularly, the evaluation team will review COMET’s performance monitoring data and prior evaluations and assessments of the project which may contain baseline and follow-on data.  Key informant and focus group interviews: Key informant and focus group interviews will be used to further investigate the results, effectiveness, and value-added of the GDA/PPP approach. Participants in these interviews will include COMETS staff, staff of private companies participating in GDAs, Ministry of Health officials at the national, district, and community level, health care workers, representatives of community-based organizations, community workers and volunteers, and project beneficiaries. While it will not be possible to randomly select participants in key informant and focus group interviews and draw a large sample, the evaluation team will ensure that the sample selected is as representative as possible to minimize bias and subjectivity to enhance the rigor of the evaluation results. USAID/Zambia will provide the evaluation team with the necessary existing performance documents, such as quarterly reports, semi- and annual program results, performance monitoring plan results, program review reports, and various project monitoring tools including a cost-benefit analysis tool that assesses the effects of HIV/AIDS interventions on workforce productivity and retention and business output. In addition, the project has a robust monitoring and evaluation system that collected data on different aspects of the program. These sources of information will be provided upon signing the contract. The evaluation team is expected to develop tools to collect and analyze information including key informant and focus group guides, data and content analysis templates, and baseline data reconstruction methodologies. C.5.3 Data Analysis The evaluation team will develop a plan to analyze qualitative and quantitative data and triangulate evidence in order to make rigorous and evidenced-based conclusions about the effectiveness of the project. The data analysis plan, to be included in the overall work plan; will specify the methodology for data analysis, including use of data analysis software. Illustrative data analysis methods include content analysis for various documents reviewed by the evaluation team, descriptive analysis for quantitative data (mean, median, standard deviation, cross-tabulations), before and after comparisons, and cost-benefit analysis. Where necessary, data will be disaggregated by gender, industry, type of intervention, type of population served, etc. Data will be compared and triangulated to make sure that the responses to each evaluation question obtained through the various data sources and collection methods point to similar findings. Annex . 5 C.5.4 Limitations of the Proposed Design and Methodology While the project has collected baseline data on some of its performance indicators, there are many key indicators that don’t have baseline data, especially those related to changes in knowledge, attitudes, and behaviors. In addition, even if there are cost-effectiveness studies conducted, it might be difficult to assess the outcomes of the HIV/AIDS workplace programs (increased worker productivity, reduced absenteeism, increased retention and business output) and quantify the direct contribution of the project to reducing the impact of HIV/AIDS in the agriculture and mining sector. Also, in the absence of a random sample selection, several biases could be introduced while selecting interview participants and sites and deciding about the size of the sample. The offeror is expected to implement a rigorous design and evaluation methodology despite all the potential data and budget limitations in order to increase the rigor and credibility of the evaluation results. C.6 DELIVERABLES 1. Evaluation Design and Methodology: Five business days after arriving in Zambia, the evaluation team shall submit an evaluation design and methodology for USAID/Zambia review and approval. The design and methodology should also include data collection tools and a data analysis plan. 2. Work plan: Two business days after approval of the evaluation design and methodology, the team shall submit the work plan for USAID/Zambia approval. 3. Summary Presentation of Findings to USAID/Zambia and stakeholders: Two business days prior to departing Zambia, the evaluation team shall present initial findings to USAID/Zambia and other stakeholders for review, comment and feedback. A PowerPoint presentation and handout (maximum of two pages) shall be prepared for the presentation to USAID and stakeholders. 4. Evaluation Report: A draft evaluation report is due five business days after the field visit is completed. Within 10 business days of receiving USAID/Zambia’s feedback to the draft report, two hard copies and one electronic (MS Word) copy of the final evaluation report are due to USAID/Zambia. The evaluation report should include the following: a. Executive summary b. Background; c. Introduction; d. Methodology; e. Findings, including lessons learned; f. Conclusions; g. Recommendations; and h. Annexes including: i. Scope of Work ii. Data collection tools iii. Key data sets including interview transcripts iv. List of key informants v. Documents consulted The evaluation report should meet the criteria for quality evaluation reports specified in Appendix I of Evaluation Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf). If USAID Zambia disagrees with any aspects of the report, the evaluation team is expected to include a section in the report describing the points of disagreement. Annex . 6 ANNEX II: EVALUATION METHODS The following Evaluation Methodology was approved by USAID/Zambia on November 7, 2012. INTRODUCTION Social Impact, Inc. (SI) has been requested by USAID/Zambia to conduct an end-of-project impact evaluation of the Community Empowerment Through Self-Reliance (COMETS) Project. COMETS was awarded as a cooperative agreement through the New Partners Initiative (NPI) to the Comprehensive HIV/AIDS Management Program (CHAMP), a private, non-profit indigenous organization established in 2002 to combat the HIV and AIDS epidemic in Zambia. COMETS began on September 19, 2008 with a completion date of September 30, 2012. The life-of-project cooperative agreement amount was $9,000,000. COMETS was designed to demonstrate the pivotal role that local organizations and public-private partnerships can play in attaining Zambia’s goals of reducing both HIV/STI transmission and the socio￾economic impact of HIV/AIDS, while increasing access to quality HIV/AIDS and STI prevention, care and treatment interventions. A core component of COMETS was programmatic consolidation of effort with USAID/Zambia’s Global Development Alliance (GDA) private sector partners in the agribusiness and mining sectors. COMETS had two overarching objectives:  To support the private sector GDA partners in strengthening their HIV workplace programs and expanding programs in the local surrounding communities,23 and  To reduce the impact of HIV/AIDS in the agribusiness and the mining sectors by addressing issues of increasing productivity, reducing absenteeism, retention of skilled employees and labor and increased business output. PURPOSE AND USES OF THE EVALUATION The purpose of this evaluation is to assess the program’s effectiveness, including identification of factors that may have enabled or constrained successful outcomes. The evaluation is intended to generate information that will be useful to policy-makers and program managers who are concerned with improving future programming that would utilize public-private sector partnerships. It is anticipated that the evaluation will be used by various stakeholders for the following purposes: 6. USAID/Zambia: To identify promising practices and areas for improvement and to inform the design of potential future HIV interventions under USAID/Zambia’s Country Development Cooperation Strategy. 7. USAID/Global Health Bureau, PEPFAR, and other partners in health: to demonstrate effectiveness of the partnership approach with the private sector and local organizations in HIV programs. 8. USAID/Policy Planning and Learning Bureau: to suggest a model for evaluating public-private partnerships. 9. Government of the Republic of Zambia (GRZ): to demonstrate the effectiveness of the public￾private partnership model as a potential methodology for a sustainable HIV response by the government. 10. CHAMP/COMETS and GDA Partners: to inform the strengthening of its partnerships and programs. 23 Under COMETS, the local community was defined as a Global Development Alliance partner community population living within the catchment area of a public rural health center (RHC) as identified by the Ministry of Health’s District Health Office. The size of such communities feeding into a RHC typically ranged between 5,000 and 15,000 individuals. Annex . 7 It is expected that USAID/Zambia will disseminate the report widely, including in the Development Experience Clearinghouse (DEC), with relevant stakeholders and project beneficiaries and may use the evaluation in the design of future projects and publications. EVALUATION SCOPE AND FRAMEWORK According to USAID guidelines, an impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy. This involves counterfactual analysis, that is, a comparison between what actually happened and what would have happened in the absence of the intervention. Although this evaluation addresses issues related to impact on beneficiaries, communities, and partner institutions, this cannot be a classic impact evaluation based on a “gold standard” experimental design format as defined by USAID guidelines, because the program was implemented strategically, targeted at specific partners and communities. In this case, the estimation of a counterfactual is not possible.24 The approach will be largely non-experimental through a systematic and comprehensive review of program outputs and outcomes on beneficiaries, using alternative methods to suggest the impact of the COMETS activities on participants. This evaluation is intended to measure quantitative and qualitative changes that have occurred in subject organizations, communities and people. Where possible, we will make pre-/post- comparisons by using pre-program data, but we have designed our methodology under the assumption that baseline data may not be available in many cases. Measurement of outcomes and/or impacts25 will include cost-benefit analyses to evaluate the financial success, quantitative and qualitative beneficiary analyses focused on the individuals in the workplace and surrounding communities, and stakeholder analyses focused on the organizational and management aspects of project performance.. The evaluation will then go further to identify successful structural, managerial, or program factors and strategies that may be worthy of replication. Thematic Analyses The Key Evaluation Questions (KQs) will be tackled from 3 methodological vantage points that address key thematic areas. These are:  Quantitative measures, including economic and cost-benefit analysis primarily focused on GDA partner organizations and beneficiary analysis primarily focused on individuals within the workplace and community  Qualitative measures, including actual and perceived changes in the impact of HIV/AIDS upon the vitality of communities and people.  Organizational measures that consider stakeholder, program, project and management structures and systems and the results of capacity building efforts. Key Evaluation Questions The evaluation will focus on seeking answers to a complex and interrelated set of KQs: 3. To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? 4. How effective has the Global Development Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? 24 USAID Evaluation Policy (2011) Evaluation: Learning from Experience. USAID, January 2011: Washington, DC. 25 Here, a distinction is being made between near-term results or “outcomes” that can be observed shortly following completion of the program, as compared to “impact” (that is, long-term and sustained change as a result of the program. Annex . 8 6. What has been the project’s contribution to reducing impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? 7. To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost-efficiency, and monitoring and evaluation? 8. Besides the USG funding contribution, what has been the additive value, if any, of the USG partnership with GDA partners? d. What has been the added value to private companies, GRZ, and local communities? e. What additional benefits have been realized in engaging Private companies that already have corporate social responsibility? f. What multiplier effects have the USG-GDA partnerships had within and outside of HIV/AIDS? SOURCES OF DATA AND DATA COLLECTION ACTIVITIES To address each of these Key Evaluation Questions, we will rely on a variety of data sources and data collection methods. The Data Collection and Analysis Matrix in Annex 2 is organized around each of the evaluation’s 5 Key Questions (KQs), and provides a description of data collection methods to be used (primarily file data or interviews). Annex 2 describes the wide variety of data to be extracted and analyzed, the sources(s) of that data, and the types of analyses that will be undertaken to inform formulation of findings and conclusions. KQ1 is concerned with capacity building and will be addressed with data and information drawn from project progress reports, COMETS impact assessment reports, key informant interviews with GDA partners and the New Partnership Initiatives assistance agency. KQ2 is concerned with effectiveness of the GDA/PPP model and will be addressed with data and information drawn from project financial statements, data from the CHAMP/COMETS Monitoring and Evaluation database, data from GDA partners’ project records, and interviews with representatives from GDA partners. KQ3 is concerned with reducing the impact of HIV/AIDS in the agribusiness and mining sectors – specifically with reference to productive areas within the workplace – and will be addressed with data and information drawn from data abstraction from CHAMP/COMETS Monitoring and Evaluation database, data from GDA partners’ Human Resources and Corporate Affairs departments as available, data from SmartCare a component of the the national HIV/AIDS information system) as well as partners’ on-site health facilities as applicable, due diligence records on GDA partners undertaken by CHAMP, COMETS internal impact assessment reports, interviews with community and partner focal points, and qualitative data from focus group discussions (FGDs) with individuals in the surrounding communities as time and logistics permit. KQ4 concerned with the effectiveness of PPP strategies will be addressed with information drawn from stakeholders, key informants, and individual and community beneficiaries. KQ5 is concerned with the additive value of USG funding and will be addressed with data and information drawn from project progress and activity reports, data from the CHAMP/COMETS Monitoring and Evaluation database for COMETS project, GDA Partners, HIV/AIDS focal persons, HR Managers, and community volunteer leaders. As described above in the approach to answering each KQ, substantial weight in this evaluation will be placed on use of data and information to be extracted from existing CHAMP and GDA files. The methodological approach of relying heavily on existing file data presents some risk in that the quantitative analyses in this evaluation will be dependent, to a large extent, on the availability/accessibility of data from CHAMP/COMETS and GDA files. The team has been assured, however, that CHAMP/COMETS will do everything possible to meet our requests for file data. Nevertheless, some information will not have been collected over the life of the project and therefore the team will do the best with the information available from CHAMP/COMETS M&E, project and partner reports, financial records, SmartCare records, and other sources of quantitative data. Our quantitative economic and beneficiary analysis will be supplemented by qualitative information that probes for objective themes Annex . 9 essential to the focus and KQs of this evaluation as well as to record subjective experiences of key stakeholders; this qualitative and quantitative derived from interviews with selected key informants including GRZ representatives (MOH and NAC), and GDA partners. Selection of GDA Partner Key Informants Given the extremely limited number of working days provided in this evaluation for data gathering, analysis, and synthesis, the team will not be able to draw a large sample of key informants from the universe of 14 GDAs. In this way we cannot assure that data is collected representatively from each of the private sector partners. It is important to note that every effort will be made to collect data from all 14 partners when possible, specifically through CHAMP/COMETS M&E and SmartCare systems as well as financial records and annual project reports. However, for more in-depth interviewing and some of the qualitative methods, it is necessary to be mindful of the availability of access to GDA Focal Points and other key staff on short notice, and accessibility considerations. (In our final report, we will briefly touch upon the implications of selecting the particular firms for this in-depth analysis.) The team has selected an approach to approximate representativeness by purposively selecting private sector partners. We will use a small purposive sample of 4 GDAs based on the following criteria to ensure some representativeness across different configurations of partner programs: - Sector: Represent both the agribusiness and mining sectors - Program Breadth: Represent GDA models that have both limited as well as extensive community outreach programs; those that do and do not have on-site health facilities; those with and without certain HIV/AIDS related services and activities - Firm Characteristics: Represent firms that have workforces of varying size - Program Longevity: Represent programs that have been in operation for different lengths of time, where possible - Geography: Represent, as time and logistics permit, firms in different parts of the country Using these criteria, two mining and two agribusiness firms have been selected. The first agribusiness firm is Zambia Sugar Company in Mazabuka, which includes workplace and surrounding community components in its program. The second firm selected is Dunavant Zambia which works closely with a network of public health facilities and covers a larger community in Southern and Eastern Provinces of Zambia. Mining partners include Mopani Mines in Kitwe with workplace programs and services in the surrounding community, and First Quantum Mines with workplace programs as well as programs covering the whole district. Whereas the approach of purposive selection of a limited number of GDAs limits the basis for drawing conclusions that represent the model across all sites, the selected GDAs have been selected because they have mature, committed programs that represent different types of programs across the GDA partners, that should be able to provide guidance on success factors as well as challenges. Where possible, visits to GDA sites will, at a minimum, include visits to CHAMP liaisons or COMETS focal points, HR departments, on-site health facilities when applicable, and CRCs. The team will attempt to visit some Rural Health Units (RHUs) but recognizes this may not be possible given the time allotted at each site (about 1 day). While on site, the team will collect quantitative and qualitative data as described in sections above; in addition, at least one Focus Group Discussion is planned with beneficiaries who are associated with a GDA workplace or community outreach programs. Other data sources Supplementary data from other authoritative sources will be obtained. A detailed literature review of other related research work and studies undertaken by independent sources will also be carried out. We will also explore cross-cutting issues in interviews with key informants. In addition to KQs Annex . 10 described, we will probe for specific themes as deemed appropriate within the context of interviews, including:  Gender – The study will inquire about existing programs that promote gender equality;  HIV/AIDS – The analysis will ensure that the plight of those affected by HIV/AIDS is also taken into account, and where possible, actions that could empower the victims are promoted such as HIV and AIDS workplace policies will be explored with informants;  Environmental Sustainability – The analysis will also consider the impact of current and proposed actions on the environment, and explore ways in which environmentally friendly practices can be promoted;  Long-Term Sustainability – This area may address overall financial or other sustainability issues related to continuing to implement and expand programs initiated and supported originally through the COMETS project. Care will be taken in the qualitative key informant interviews to systematically document discussions. A Structured Key Informant Interview (SKII) data gathering instrument has been designed and will be used and elaborated upon by the team to create file notes that will be appended to the evaluation report (see Annex 3). DATA ANALYSIS PROCESS The Data Collection and Analysis Matrix in Annex 2 provides an overview of how different data and analysis processes are intended to lead to concrete findings, conclusions and recommendations that are responsive to the evaluation’s Key Questions. What follows is a more detailed discussion of the various types of data analysis to be undertaken. Economic Analysis The economic analysis is grounded in a thorough understanding of USAID’s GDA approach to private sector engagement, and of PPPs more generally. Data permitting, baseline health data prior to COMETS for GDA catchments is captured and related to post-COMETS data in same catchments. It is important to note that CHAMP/COMETS often has not captured key baseline data, and in the event that partners are not able or not willing to share pre-program data, our ability to make pre-/post- comparisons will be very limited. Inputs by USG and partners/stakeholders are related to performance change permitting cost-benefit or other analysis of beneficiary impact. Company performance impact is then demonstrated by analyzing COMETS performance data with company and GDA-specific HR data. Finally, the additionally and leverage of COMETS is demonstrated by capturing unplanned new participation in COMETS or COMETS-like activity. The process undertaken for the economic analysis is detailed in Table 1 below. Annex . 11 Table 1. Economic Analysis Cascade Task Approach Strategy Data Sources SOW KQ and detail Analysis Schedule Understand GDA approach -PPP literature review -USAID GDA review -COMETS management review -GDA management review -benchmark GDA approach vs. select PPP -relate analysis to USAID GDA policy -PS and GDA management interviews -literature -USAID documentation -COMETS /industry management -project work plans, MOUs, strategic planning KQ 1-5; emphasis on KQ 2,3,4 Background for analysis 6 Nov Capture baseline health data Link COMETS inputs by stakeholder to results Link inputs to beneficiary impact Understand company contribution impact Demonstrate GDA benefit to company performance Demonstrate leverage and additionality of COMETS/GDA model Compile and write -published and unpublished MOH data -COMETS M&E reporting -Company baselines -SHARE data -financial and performance data capture -merge cost and performance data with beneficiary data -capture selected company input and performance data Understand partner/stakeholder participation beyond COMETS work plan but attributable in part to COMETs -extract / compile data from sources -create data profile/baselines for GDA catchments -extract and align COMETS costs to annual/overall performance vs. targets -data permitting, analyze unique GDA partner impact Baseline to end of project HR data: absenteeism; retention, productivity, etc. Identify non-GDA partners adding value and quantify national, M&E, district and company data -COMETS M&E, financial and narrative reporting -GDA partner input data -prior data sources Company COMETS data and interviews Company HR data; Anecdotal reports, new partner activity, in-kind contributions by non￾GDA partners KQ 3,4 Primarily KQ 3,4,5 KQ 2,3,4 KQ 2,3,4 KQ 4; KQ 5 Background for analysis CB analysis; unit costing disaggregated by GDA partner/site as possible Comparative CB analysis by GDA Comparative contribution to results by GDA Quantify Quantify 6 Nov 9 Nov 9 Nov 9 Nov 11 Nov 11 Nov 13 Nov Annex . - 12 - Quantitative Beneficiary Analysis Where possible, quasi-experimental methods will be used in carrying out analyses focused on the impacts of the COMETS program on important workplace-related and community-level outputs, and outcomes pending the availability of data from CHAMP and private sector partners. These methods very specifically address Key Evaluation Question 3, focusing on productive aspects of the impact of the COMETS program. Specifically, this part of the evaluation will look at absenteeism, productivity, and retention among employees at partner firms, while also attempting to link that information with key data including participating in workplace programs as well as health data from partner sites. In this way, the objective will be not only to document changes in the productive activities over the life-course of the COMETS program but to link that data with the expansion and participation in the programs supported by COMETS. This analysis is very sensitive to the availability of data, and the level of robustness will be directly related to the amount and quality of data that can be obtained in these areas from CHAMP/COMETS as well as partner firms. Pending the availability of data available through the CHAMP Monitoring and Evaluation department (“CHAMP M&E”), the quantitative beneficiary analysis will consist of 2 main components: (1) Comprehensive Output Aggregation of COMETS Outputs by private sector partner (2) Statistical Analysis where data is available and in sufficiently good quality Overall, data will be abstracted from CHAMP M&E as well as GDA partners (including Human Resources departments, and health facilities on-site), and any other select interviews or data abstraction deemed necessary over the course of data collection. This may include further document review from CHAMP records including internal impact assessment summaries as well as documents detailing partner activities as part of due diligence process at partnership initiation. Comprehensive Output Aggregation: Comprehensive Output Aggregation will occur using data abstracted from CHAMP M&E, pending the availability of data. The intent is that a table will be produced that can correspond to each partner, as well as COMETS as a whole, for each fiscal year of the project, following the template table shell presented below in Table 2. Table 2. Comprehensive Output Aggregation GDA Partner Summary of Partner Activities COMETS Strategic Objective(s) Indicator(s) Period Active Workplace Output(s) Community Output(s) Example: Dunavant (There will be 14 GDA partners) EX: List of activities included within COMETS at site, such as MHU, VCT, OVC register, on-site clinic, “One Man Can” or “Need to Know” etc. Reference to any of the COMETS Strategic Objectives (1 through 5) EX: indicator such as % of population tested through VCT (Fiscal Year) EX: output such as % of employees tested through VCT EX: output such as % of surrounding community tested through VCT Further information will be added as applicable from the documents referenced above. This may include information on potential impact on the community including but not limited to: Empowerment through access to services  Access to information through use of technology  Volunteer health workers trained as part of COMETS project  Counseling & Testing services  Referral network through ICT resources/networks including 990/CRC  Unintended consequences (positive or negative)  Spillovers from workplace programs into community (information, education) Annex . - 13 - Statistical Analysis: Pending the availability from CHAMP and private sector partners, statistical analysis on the impact of the COMETS activities on employees and community beneficiaries may be possible. This is likely to focus specifically on workplace impacts, such as the project objectives to reduce absenteeism, and increase productivity and retention within the workplace. It is important to reiterate that this evaluation is not able to take advantage of a program structured or implemented specifically with statistical analysis in mind. In this way, it is not relevant to calculate sample size. We can, however, plan specific analysis with a view to generating correlational evidence to link COMETS activities with key outcomes. The combination of other quantitative and qualitative data will allow us to get closer to making direct links that approximate causal relationships, but it is not possible in this evaluation, given the way it was implemented, the ex-post nature of the entire evaluation, the lack of truly robust baseline data, and the compressed data collection time-frame, to complete a comprehensive statistical analysis that allows for causal inference. Data for these statistical analyses will be derived from partner HR departments, on-site health facilities, and other focal persons such as Corporate Affairs managers as applicable, who will be alerted to our arrival through CHAMP. These departments within the partner firms have previously worked with CHAMP on similar project assessments using the same data we are requesting, and therefore are likely to have available and share with our team the necessary data for this component. However, the availability of data is not guaranteed and therefore the full scope of statistical analysis that can be done will still depend on the amount and quality of data available. Stakeholder Analysis & Qualitative Beneficiary Analysis Our aim will be to fully capture the perceived qualitative benefits accruing to the partners, communities and people associated with the GDA model and COMETS project through a focused study of industries and firms, as well as community organization as available and as time permits. These will be complemented by a qualitative analysis of both internal and external factors that affect the capabilities of firms. A focused literature review of other related research work and studies undertaken by independent sources will also be carried out as needed and if critical to understanding and contextualizing results (for example, the Zambia Demographic and Health Survey). Then we will conduct interviews with key persons as well as plan at least one focus group discussion in a partner’s surrounding community. This line of inquiry will substantially address KQs 2 and 3. This component will be guided by the following categories of questions, customized for each interview or FGD to maximize the utility of responses:  The cumulative catchment population for the Global Development Alliance (GDA)/private sector partners  Description of work place programs that are currently in place under the COMETS project.  Has the COMETS project effectively supported GDA/private sector partners to enhance and strengthen their HIV workplace programs?  Are the project activities successful? Numbers of people served and other relevant data, if available.  Has the project provided supportive supervision to ensure quality service delivery and adherence to national standards and guidelines?  Has the use of HIV and AIDS services increased in work place programs?  Are mobile HIV units (MHU) reaching communities and are services expanding as a result of these MHU? What types of services are the MHU offering? Are they adequately staffed and stocked? How many clients do they reach each month/district or community? Organizational & Management Capacity Building Analysis This analysis will be primarily focused on the perceived qualitative benefits accruing to the partner organizations in their ability to mount, manage, and sustain effective programs and interventions, emphasizing benefits that can be attributed to GDA/PPP, and COMET methodology and approach. In particular, management capacity building activities that occurred within the scope of COMETS, both Annex . - 14 - capacity building technical assistance received by COMETS, and provided by COMETS to partners, will be examined. This line of inquiry will substantially address KQs 1 and 5. The principal tool for gathering information will be the structured interviews with HIV and AIDS Focal Persons, senior representatives from the 4 identified GDA organizations and any surrounding community-based partners. Supplementary data from other authoritative sources will also be obtained (for example, USAID’s New Partnership Initiatives management assessments). A detailed literature review of other related research work and studies undertaken by external sources will also be carried out (for example, USAID’s New Partnership Initiatives assessments). The effect of various configurations of the GDA/PPP model will also be examined because there appears to be great variation in the organizational and community environments in which GDA programs have been mounted. Two agribusinesses have been selected26 for on-site visits during this evaluation that offer contrasting examples of how a GDA responded to and utilized the opportunities presented through USAID support and COMETS assistance. Zambia Sugar Company in Mazabuka not only worked with its own employees but also actively extended outreach into the community at-large. The second firm selected, Dunavant Zambia, works extensively with and through public health facilities and covers a larger community in Southern and Eastern Provinces of Zambia. Mopani Mines in Kitwe has undertaken showcase workplace programs and services in the surrounding community, and First Quantum Mines not only has active workplace programs, but also reaches the whole district and beyond. Examples of critical questions that might be posed to COMETS and GDA/PPP partners include:  What are your overall impressions of the NPI technical assistance component of the program?  What do you believe was the underlying rationale, and did NPI serve to strengthen program management capacity of COMETS? Please provide examples and details.  Was COMETS able to transfer new organizational or program management learning from its own NPI capacity building experiences to its GDA/PPP partners? How did this work? Give examples? How could this transfer (if applicable) be measured?  Have new capacity building knowledge and skills invested in GDAs had a cascade effect on the program management capacity of community partners (for instance in the area of strategic planning and budgeting)?  Do community partners perceive that their program management capacities and effectiveness have been improved through association with GDA programs? COMETS? Examples?  What factors enhanced or constrained the GDA or community partner in being able to build internal program management capacity (for instance in the area of resource management)?  How could COMETS have been used to better advantage in assisting GDA and community partners to strengthen program management performance? DATA SYNTHESIS AND TRIANGULATION All components of the COMETS project evaluation will be linked together during and after data collection in order to triangulate findings, synthesize results, and formulate cohesive, policy-relevant conclusions and recommendations. For example, the quantitative beneficiary analysis will be linked with other sections of the evaluation, namely the economic analysis as well as the qualitative beneficiary and 26 Selection of sites to visit was derived from several factors including (in the judgment of CHAMP/COMETS and USAID): the GDA’s level of involvement in the COMETS program; variations in configuration of their programs and commitment of GDA management to PPP strategies; and geographic accessibility of the evaluation Team to GDA operations and communities. Annex . - 15 - stakeholder analyses. The estimates of impact form an important component of the benefits that are included in a cost-benefit analysis (CBA). For example, this can include figures such as employee absentee time averted (and therefore directed toward productive activity) due to the COMETS program. Triangulation will also take place with qualitative components. For example, if absenteeism in quantitative data is observed, we will compare this to responses within interviews conducted among employees of private sector partners and community members; this could include perceptions of sickness and absenteeism in the community with respect to participation in COMETS programs. Given the short timeframe of the evaluation, it is not possible to carry out all possible comprehensive analysis of project impact. This process of data synthesis and recommendation development may therefore also include some recommendations in terms of future analysis that may be undertaken in possible, hinging on the future support and policy interest. REPORTING Following the fieldwork, the evaluation team will prepare and deliver a presentation to USAID/Zambia covering activities involving information sharing, discussions, team analyses to date and consolidation of evaluative data and information into formulation of preliminary findings, conclusions and recommendations as applicable. Based on feedback from the presentation, the team will draft a high￾quality evaluation report. In response to USAID specifications, a tentative outline of the evaluation final report is included in Annex 4. This outline will be expanded and become more detailed during the final report-drafting phase. WORK PLAN AND TIMELINE Below we describe the major events and timeline for activities to be completed during the evaluation in Weeks 1 through 9, followed by an Annexed Gantt chart (Annex 1). WEEK 1 Team Planning Meeting SI HQ, Key Personnel, USAID/Zambia COR WEEK 2 MAJOR EVENTS International Team Members Arrival: 29 October Initial Meeting with the Team’s Logistics Specialist: 29 October USAID/Zambia Internal Team Briefing Meeting: 29 October USAID/Zambia In-briefing: 30 October Agenda included: USAID’s general expectations; elaboration of the Key Questions (including USAID’s rationale in selecting those specific KQs to focus on in the evaluation; identification of particularly critical issues; provide guidance on key informants; and provide clarification on questions from the Team CHAMP/COMETS In-briefing: Wednesday, 31 October Follow-Up Data Abstraction Meetings at CHAMP/COMETS Offices: 1 and 2 November Team development and finalization of the Draft Evaluation Methodology and Work Plan: Draft to SI headquarters by OOB (EST) Friday, 2 November Submission to USAID by OOB (CAT) Monday, 5 November USAID approval/comments expected by COB (CAT) Tuesday, 6 November WEEK 3 MAJOR EVENTS Meetings with Stakeholders and Key Informants Including MOH, NAC, FHI 360, and others suggested by USAID and COMETS Field Site Visits: 6-7 November to 2 GDA agribusiness partners, (Zambia Sugar, and Dunavant) at Mazabuka, including community partners and beneficiaries. Field Site Visits: 8-9 November to 2 mining partners (Mopani at Kitwe, and First Quantum at Ndola), including community partners, beneficiaries, and the former COMETS GDA Chairman. Annex . - 16 - On-going data collection, data analysis and synthesis, team triangulation, and report drafting WEEK 4 MAJOR EVENTS Data synthesis and consolidation of draft report sections Preparation of PowerPoint presentation for USAID/Stakeholder Out-briefing USAID/Stakeholder Out-briefing: 14 November Team departure from Lusaka: 17 November WEEKS 6-9 FINAL DELIVERABLES  Draft Final Report submitted to USAID by 26 November  USAID review comments by 7 December  Final Report submission by 21 December Annex . 17 ANNEX 1. Work Plan Gantt Chart October 29 – December 21, 2012 ACTIVITIES 1 2 3 4 5 6 7 8 9 Initial Team Planning Meeting at SI HQ, Arlington, VA, USA Team Travel to Lusaka, Zambia (29 Oct) In-Brief with USAID on plan, approach and information resources In-Brief with CHAMP on plan, approach and information resources Prepare Draft Evaluation Methodology and Work Plan Finalize Data Collection Instruments USAID Review Draft Evaluation Methodology and Work Plan (5 Nov) Submit Final Evaluation Methodology and Work Plan (after 6 Nov) D Site Visits Begin Analyzing Data and Prepare Presentation for USAID/Zambia Out-brief and Presentation with USAID (14 Nov) D Team Departs Field (17 Nov) Write and submit draft final report (26 Nov) D Revise report incorporating USAID comments (after 7 Dec) Submit Final Report to USAID (21 Dec) D For a full list of corresponding dates, see workplan details above in report Annex . 18 ANNEX 2. Data Collection and Analysis Matrix Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 1. To what extent has COMETS built the capacity of the GDA partners to strengthen, sustain and expand their HIV workplace programs? Comparative Document review  Project progress reports  COMETS impact assessment reports N/A Compare observed and reported outputs and outcomes with indicator targets (indicators include PEPFAR as well as CHAMP indicators Interviews to understand challenges in meeting targets and revisions to targets. Semi-structured interviews  Key informants from private sector GDA partners  Key informants from institutional partners (e.g. health facilities, program managers) Purposive 2. How effective has the Global Alliance/Public Private Partnership model been in terms of the implementation of project activities and achievement of expected results? Comparative/ Analytic Document Review Data Abstraction  Project financial statements  COMETS monitoring and evaluation database  Data from GDA partners’ project records N/A Conduct cost-benefit analysis for COMETS programs. To the extent possible, benchmark cost effectiveness against similar projects. Interviews to understand perception of GDA model; challenges in implementation; insight into future scale-up Semi-structured Interviews  Key informants from GDA partners Purposive Annex . 19 Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 3. What has been the project’s contribution to reducing impact of HIV/AIDS in the agribusiness and the mining sectors? Specifically, to what extent has the project contributed to increasing productivity, reducing absenteeism, increasing retention of skilled employees and labor and increasing business output? Analytic/ Comparative Document review Data Abstraction Statistical Analysis (as applicable)  Data abstraction from CHAMP Monitoring and Evaluation database for COMETS project  Data from GDA partners  Data from health facilities on-site at GDA partners as applicable (for both participants and non-participants in COMETS projects for spillover effects, where data available)  Due diligence records on GDA partners undertaken by CHAMP  COMETS internal impact assessment reports N/A (Targeted M&E information on GDA partners) Use quantitative analysis to determine quantifiable effects of the COMETS program on relevant workplace￾related outcomes at the individual level among employees of GDA partners affected by workplace programs as well as individuals in surrounding communities affected by outreach activities; Triangulate with content analysis of beneficiary interviews. Use qualitative methods followed by content analysis of interviews to uncover themes in contributions of the COMETS project. Semi-structured interviews Data from GDA Partners activity reports, M& E data, key informants, beneficiaries Purposive Annex . 20 Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis 4. To what extent has USAID’s approach of working directly with local implementing organizations and private sector partners been effective? What have been its strengths and challenges with regards to management, procurement, communications, cost￾efficiency, and monitoring and evaluation? Comparative/ Analytic Semi-structured interviews  Stakeholders, key informants, beneficiaries Purposive Content analysis of key informant interviews to discover likelihood of sustainability of project results. 5. Besides the USG funding contribution, what has been the additive value, if any, of the USG partnership with GDA partners? g. What has been the added value to private companies, Comparative Document review  Project progress and activity reports  Data from CHAMP Monitoring and Evaluation database for COMETS project N/A Content analysis of key informant and beneficiary interview data. Triangulate with observations from field visits to project sites, quantitative and qualitative beneficiary analyses. Annex . 21 Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data Analysis GRZ, and local communities? h. What additional benefits have been realized in engaging Private companies that already have corporate social responsibility? i. What multiplier effects have the USG￾GDA partnerships had within and outside of HIV/AIDS? Semi-structured interviews  GDA Partners, HIV AIDS Focal persons,  HR Managers  Community Volunteer leaders Purposive