Evaluation of the AVERT SOCIETY PROJECT Final report FEBRUARY 2006 This publication was produced for review by the United States Agency for International Development. It was prepared by Chemonics International. Evaluation of the AVERT SOCIETY PROJECT Final report GHS-I-00-03-00025-00, Task Order #05 The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The Avert Society project evaluation team is grateful to USAID/India USAID Office of Population, Health and Nutrition (PHN) colleagues for their support and interest throughout this assignment. First and foremost we thank Sampath Kumar for his enthusiasm, responsiveness and attention to detail, which led to an assignment pleasingly free of administrative and strategic hang-ups. We also benefited from the probing insights of Robert Clay, Meri Sinnitt, Janet Hayman and, especially, Sanjay Kapur. In the planning and implementation of this evaluation we were ably and flexibly supported by Lara Hensley of the International Health Group of Chemonics International, and benefited as well from technical inputs of Bill Slocum, Kathryn Bacon, Richard Pollard, and Laurie Chamberlain. We also appreciated the timely back-up provided by the New Delhi and New York offices of the Population Council, co-sponsor with Chemonics of the evaluation. We offer special thanks to Avert Society Project Director Dr. S.M. Sapatnekar, Associate Project Director Vishwanath Koliwad, and their entire staff for warmly welcoming the evaluation team to Mumbai, and for their responsiveness to us throughout the assignment. Despite work pressures inherent in the continued scaling up of their program, several Avert Society technical staff members took time to accompany the evaluation team on its field visits, some for several days. This greatly enhanced opportunities for us to fully comprehend the complexity of the program. And a special word of thanks is due to Milan Godse for her flawless handling of our logistical arrangements. Finally, we offer words of deep admiration to NGO administrators and field workers in districts of Maharashtra visited by the team, who welcomed us into their midst, put up with our questions, and proudly showed us their work. We were awed by the devotion that they bring to this essential, indeed fateful mission of preventing HIV from doing greater damage to the health of Indian society than it has already, and of caring for those already afflicted. Eliot T. Putnam, Jr. D.K. Bose G.D. Ravindran, M.D. Ravi Verma February 2006 CONTENTS i CONTENTS ACKNOWLEDGEMENTS ACRONYMS.............................................................................................iii EXECUTIVE SUMMARY ......................................................................... v I. INTRODUCTION .................................................................................. 1 I.A. HIV and AIDS in India and Maharashtra State.............................. 1 I.B. Government and USAID response ................................................ 1 I.C. The Avert Society - from early difficulties to new resolve ............. 2 II. EVALUATION METHODOLOGY ........................................................ 5 II.A. NGO partner questionnaire........................................................... 5 II.B. Fieldwork coverage....................................................................... 6 III. FINDINGS AND RECOMMENDATION.............................................. 7 III.A. Avert Society program strategy ................................................... 7 III.B. HIV and AIDS prevention through targeted interventions ......... 10 III.C. Workplace interventions ............................................................ 14 III.D. Care and Support services ........................................................ 15 III.E. Communication .......................................................................... 21 III.F Capacity building......................................................................... 23 III.G. Research / M&E......................................................................... 25 III.H. Condom promotion and social marketing.................................. 28 IV. PROJECT MANAGEMENT.............................................................. 31 V. PROJECT GOVERNANCE............................................................... 35 VI. OVERALL RECOMMENDATION .................................................... 37 Annexes A. Persons and organizations contacted ............................................ 39 B. Questionnaire circulated to NGO partners ..................................... 43 C. Avert Society evaluation schedule.................................................. 47 D. Evaluation Scope of Work............................................................... 53 ACRONYMS iii ACRONYMS ABC approach Abstinence, Being faithful, Correct and consistent condom use APAC AIDS Prevention and Control Project (Tamil Nadu State) APD Associate project director ART Anti-retroviral therapy ARV Anti-retrovirals (drugs) BCC Behavior change communication BSS Behavioral surveillance survey C&S Care and support CCC Community care center CSW Commercial sex worker CSM Condom social marketing DHS Department of Health Services DIC Drop-in center ESRM Experience-sharing review meeting (with partner NGOs) FHI Family Health International GB Governing Board (of the Avert Society) GOI Government of India HBC Home-based care HIV+ HIV positive HLFPPT Hindustan Latex Family Planning Promotion Trust IEC Information, education and communication (materials) IPC Interpersonal communication JHU Johns Hopkins University (Center for Communications Programs) M&E Monitoring and Evaluation MDACS Mumbai District AIDS Control Society MIS Management information systems MOU Memorandum of understanding MSACS Maharashtra State AIDS Control Society MSM Men who have sex with men MSW Master of Social Work MTR Monthly technical report NACO National AIDS Control Organization iv ACRONYMS NACP-2 Second National AIDS Control Plan NGO Nongovernmental organization OI Opportunistic infection OR Operations research ORW Outreach worker PD Project director PHN USAID Office of Population, Health and Nutrition PLHA Person living with HIV and AIDS PSLV Participatory site learning visit (to NGO partners) STAPI Sosva Training and Promotion Institute STI Sexually transmitted infection TI Targeted intervention TISS Tata Institute of Social Studies TOT Training of Trainers VCTC Voluntary counseling and testing center VHS Voluntary Health Services (parent NGO of APAC) WPI Workplace intervention EVALUATIONOF THE AVERT SOCIETY PROJECT v EXECUTIVE SUMMARY The Avert Society project was launched in 2001, under a $41.5 million bilateral agreement between USAID and the Government of India. It was designed to take a lead role in reducing the impact of HIV and AIDS in the State of Maharashtra, where statistics on the expansion of the epidemic are the most ominous in all of India. Under the aegis of the National AIDS Control Organization (NACO), and in collaboration with two public sector stakeholders, the Maharashtra State AIDS Control Society (MSACS) and the Mumbai District AIDS Control Society (MDACS), Avert has sought to build capacity of indigenous NGOs to reduce transmission and mitigate the impact of HIV and AIDS. In so doing, it modeled itself in large measure after the successful USAID-funded AIDS Prevention and Control Project (APAC) in Tamil Nadu. As of October 2005, the Avert Society had completed four years of implementing HIV prevention and care and support (C&S) activities in Maharashtra through NGO partners and other stakeholders. However, these years were marked by significant growing pains, such that by that date only $4.8 million of the $21.5 million earmarked for disbursement through NACO for project implementation had been spent. Funds available from the $20 million technical component, earmarked for direct support for state-level activities, had been similarly underutilized. This meant that, although the project is scheduled to come to an end in September 2006, enough money remains from the original authorization to support project activity for several more years, should USAID so decide. Reasons for the project's early difficulties included the absence of a parent organization to guide its operations, persistent weakness in senior management that resulted in high staff turnover, and a funding crisis in 2004 and early 2005 due to non-release by NACO of project funds. Since that time, however, the Avert Society has been under new management, and has been fully focused on building up both the scope and reputation of the project. A young, motivated technical staff has been recruited; the Society's financial house has been put in order, to such an extent that NACO has doubled the size of the project's revolving fund; new management and operational systems are in place; and the roster of Avert partner NGOs is now increasing steadily. A team recruited by USAID through Chemonics International and the Population Council to undertake an evaluation of the Avert Society in November and December vi FINAL REPORT 2005 found an energized agency with a clear sense of mission. After circulating an anonymous questionnaire to gather information and impressions from Avert partner NGOs, the team spent a month in Maharashtra talking at length with Avert staff, meeting with MDACS, MSACS and other stakeholders, and visiting almost 75 percent of Avert NGO partners throughout the state. The Avert program strategy involves targeted interventions (TI) among groups at high risk of HIV transmission in seven selected districts of Maharashtra, and a range of C&S service initiatives in these and other districts, all implemented through NGO partners selected after a rigorous screening and training process. It also operates under a state￾level mandate to pursue cross-cutting activities in capacity building, research, and communication. The evaluation team found this strategy to be conceptually sound, well positioned to respond to new opportunities, and in synergy with other programs and partners. It did however recommend a moratorium (suggested for six months) on most new partnerships once current targets are reached in March 2006, to allow time for new initiatives to be well grounded and to strengthen technical support in several areas. It also urged constant monitoring, primarily through the Avert Governing Board (GB), of coordination of stakeholder activities in the state (notably those of MDACS and MSACS, who are GB members) so as to maximize use of resources and avoid duplication. This report looks at Avert's accomplishments and technical capacity in: • targeted interventions, including workplace interventions • care and support services, including community care centers, drop-in centers, STI services and other approaches to mitigating impact on people living with HIV and AIDS • orienting communication strategy to behavior change communication • capacity building of partner NGOs, with a renewed emphasis on counseling • broadening the research agenda and linking it closely with monitoring and evaluation systems and philosophy • condom promotion and social marketing It concludes that in most areas Avert has set high standards and is having significant impact at district and community levels. At the same time there are a number of areas for growth and strengthening, such as: broadening contacts with high-risk groups while at the same time focusing on behaviors rather than people; enhancing support for key frontline workers, such as peer educators; developing more effective referral networks for testing and STI services; reviewing and updating training curricula. Each section dealing with a technical area of Avert activity concludes with a series of targeted recommendations. The evaluation team also recommends a number of personnel additions and adjustments that will, in its view, greatly enhance efforts to strengthen overall technical capacity and output. On the subject of project management, the evaluation team sees an organization with a clear vision and commitment, that has successfully, after some difficult times, established EVALUATIONOF THE AVERT SOCIETY PROJECT vii itself as a stand-alone agency of considerable potential, one that is much appreciated and respected by its grantees. But that potential can only be fully realized if attention is paid more closely to the Avert organizational culture. Here the evaluators found a significant need, and desire, for team building to bridge internal divides and create a more cohesive sense of mission, especially between the technical and financial management sides of the office. It urges USAID to encourage and facilitate essential team-building initiatives. Perhaps most importantly of all, there is a leadership gap that needs to be filled, in terms of how the Project Director (PD) perceives his role and interacts with his staff. The report discusses this at some length, with options. It urges USAID to take a firm, proactive role in working with the PD on an ongoing basis to clarify his perceptions and to work on aspects of his leadership style, to include formal leadership training as needed. The evaluation team also urges USAID to use the leverage inherent in its role as Vice Chair of the Avert Society Governing Board (and principle project funder) to see that GB oversight is responsive to organizational needs, rather than an obstacle to its functioning. Steps have been taken lately to streamline GB functions, which are to be encouraged and expanded when needed. The Avert Society evaluation team is fully confident that the issues it raises in its report can and will be resolved, and that Avert will continue to grow in strength and reputation. It therefore highly recommends that USAID make whatever arrangements are necessary (presumably an unfunded extension) to see that the work of the Society continues well beyond its scheduled termination date of September 2006. Given its “burn rate” to date and the size of the original authorization, Avert should be able to continue vigorous operations for three to four more years. This gives it an excellent chance to make a major contribution to slowing the advance of HIV and AIDS in the bellwether State of Maharashtra. EVALUATION OF THE AVERT SOCIETY PROJECT 1 I. INTRODUCTION IA. HIV AND AIDS IN INDIA AND MAHARASHTRA STATE Current statistics about the AIDS pandemic reveal that 10 percent of people living with HIV and AIDS in the world are in India. As of early 2005, with an estimated 5.1 million people infected with HIV, India ranked second among all countries in total numbers of people living with the virus. By any measure, the pandemic represents an acute and growing challenge for Indian public health systems. Six Indian states – Andra Pradesh, Karnataka, Maharashtra, Manipur, Nagaland and Tamil Nadu – have an HIV prevalence of over one percent among pregnant women, a proxy for the general population. Among these, the situation in the State of Maharashtra, considered the epicenter of the epidemic in India, is the most ominous. Over 1 million HIV+ people live in the state, 21 percent of all cases nationwide. Antenatal HIV prevalence increased from 1 percent in 1993 to 1.25 percent in 2003. An epidemic that first reached crisis proportions among “high risk” groups in Mumbai and other urban areas has now spread to the general population, with 60 percent of new cases occurring in rural areas of the state. Meanwhile, the stigma attached to people living with HIV and AIDS (PLHA) remains strong in India, as in other countries, hindering growth of awareness and action. IB. GOVERNMENT AND USAID RESPONSE All HIV and AIDS prevention and care initiatives in India are undertaken in concert with the goals of the Government of India's (GOI) Second National AIDS Control Plan (NACP-2), which are to reduce the spread of HIV in India and strengthen the country's response on a long-term basis. Leadership of the national response rests with the National AIDS Control Organization (NACO). A key NACO strategy has been to establish “AIDS Control Societies” in high prevalence states, as the principle vehicle through which to combat the epidemic. Early on, the Government of Maharashtra established the Maharashtra State AIDS Control Society (MSACS) to implement a range of HIV prevention activities throughout the state, except in Mumbai. Because of the size of its population, and the numbers of high￾risk groups within its limits, a separate Mumbai District AIDS Control Society (MDACS) was set up to initiate prevention programs in the city/district itself. Together, MSACS and MDACS spearhead the public sector's response to the challenge of HIV and AIDS in the state. USAID/India's bilateral support to the GOI for combating the epidemic seeks to complement NACO strategy by stabilizing HIV prevalence in the general population by reducing 2 FINAL REPORT HIV prevalence in at-risk groups and improving care and support for those infected and affected by HIV and AIDS. 1 USAID supports two large, state-specific projects that address this objective, largely through broad-based support of nongovernmental organizations (NGOs.) Since 1995, under a $10 million tripartite agreement signed with NACO and Voluntary Health Services (VHS) of Chennai, USAID has funded the AIDS Prevention and Control (APAC) project in Tamil Nadu, another “high prevalence” state. APAC was designed to prevent the spread of HIV and AIDS by strengthening the capacity of Indian NGOs to undertake prevention activities among high-risk groups. A mid-term evaluation of APAC in 2000 found that the project was successfully meeting this objective, and that it could serve as an important model for the rest of the country.2 Renewed and refunded since that time, the project continues its leadership role to this day. The initiative most directly modeled after APAC was the Avert Society, for which a seven-year, $41.5 million bilateral agreement was signed between USAID and the GOI in 1999. Ultimately launched in November 2001, Avert was designed to reduce the impact of HIV and AIDS in the State of Maharashtra. As with APAC, its primary mechanism for doing so was through building capacity of a wide range of indigenous NGOs. Unlike APAC, which had VHS as a “parent” agency to guide the project's operations in Tamil Nadu, the Avert Society found no such Maharashtran umbrella organization. Rather, it has existed as an independent entity, overseen by a Governing Board (GB) that is responsible for monitoring achievement of project goals, policy guidance and major funding decisions. Funding for Avert from USAID was divided into two components: (1) a $21.5 million agreement between USAID and NACO for project implementation, focused primarily in seven target districts; and (2) a $20 million “technical component” for direct disbursement by USAID, intended largely for scaling up state-level activities such as communication and contraceptive social marketing (CSM). A close working partnership between MSACS, MDACS and Avert was envisaged, so as to achieve synergy and avoid duplication in meeting the enormous challenge of the HIV epidemic in Maharashtra. IC. THE AVERT SOCIETY – FROM EARLY DIFFICULTIES TO NEW RESOLVE At the time of this evaluation, the Avert Society project had completed four years of implementing HIV prevention and care activities in Maharashtra through NGOs and other stakeholders. Over that period the project experienced significant growing pains, such that as of October 2005, only $4.8 million of the $21.5 million earmarked for disbursement through NACO for project implementation had been spent, and funds available from the technical component had been similarly underutilized. This meant that, although the seven-year life of the project was to come to an end in September 2006, enough money remained from the original project authorization to make possible an unfunded extension of several years in duration, should USAID so decide. Several reasons for the project's early difficulties can be cited, including: 1 From "HIV/AIDS Strategic Plan for India, 2003-2007", USAID/India 2 See "Midterm Evaluation, AIDS Prevention and Control Project (APAC)", submitted by the Synergy Project to USAID/India, May 2000 EVALUATION OF THE AVERT SOCIETY PROJECT 3 • lack of a parent organization (see above) to help the fledgling project over the rough spots, especially in its early stages • serious management shortcomings, especially under Avert's first two project directors, and persistent staff turnover • a major financial crisis in 2004 and early 2005 due to non-release of resources for implementation of project activities by NACO to the Avert revolving fund • inadequately developed management systems, and a growing backlog of uncleared audits Since early 2005, the Avert Society has been under new management, one that seems fully focused on the future and has recruited a young, motivated technical staff. All audits have been cleared. A new Avert management systems manual will be operationalized in early 2006. NACO's release of funds to Avert is now timely and trouble-free. Indeed, NACO has agreed to double the size of the project's revolving fund, giving Avert financial managers much needed flexibility in planning disbursements. The roster of Avert partner NGOs had grown from around 35 in September 2005 to 54 at the time of this evaluation, and is projected to number close to 100 by March 2006. As a member of the staff said to the evaluation team, over the past year Avert has been both “catching up and scaling up”. The evaluation team found a palpable energy about the project, and a refreshing sense of mission.. EVALUATION OF THE AVERT SOCIETY PROJECT 5 II. EVALUATION APPROACH AND METHODOLOGY Despite its problematic past, USAID/India has maintained support for the Avert Society project longer than might have been the case elsewhere, primarily because the urgency of stemming the tide of HIV infection in Maharashtra is so great. Thus, the scheduling of this evaluation was timely, in that it afforded the opportunity to examine the project at a promising yet critical juncture, assess Avert's capacity to move forward with confidence and effectiveness, and thereby determine whether USAID’s, and NACO’s, trust in the project's future is justified. For these reasons, the evaluation team took a forward-looking approach to its data gathering and analysis, using a variety of methods for obtaining information. It felt there was benefit in revisiting past problems only to the extent necessary to learn lessons that would inform future decision-making, judge the quality of current interventions, and identify technical and management areas in need of strengthening. The four members of the team brought to the task extensive experience in HIV and AIDS program management and research, expertise in comprehensive approaches to behavior change communication (BCC), and a high degree of technical competence in quality assurance and the provision of care and support services. (See Annex D for the evaluation Scope of Work.) II.A. NGO PARTNER QUESTIONNAIRE Before assembling in-country, and with the assistance of the Population Council office in New Delhi, the team drafted and circulated a short, anonymous questionnaire to NGO partners of the Avert Society and other stakeholders. The questionnaire (see Annex B) asked respondents to describe the work for which they received support from Avert, the different elements of that support, and the extent to which it had or had not met their expectations. It invited them to look to the future, and to suggest ways in which Avert might help them strengthen and expand their mission in the months and years ahead. The purpose of the questionnaire was to give the team a snapshot of the project, from sub-grantees’ perspective, at this point in its life, and clues as to what to look for in field visits. Responses received from NGO partners spoke positively of the Avert Society's methodical, “hands-on” approach to the development of partnerships, its strong and consistent emphasis on capacity building, the technical competence of the Avert team and its responsiveness to their needs. They also highlighted areas of concern, such as inconsistent communications, delays in release of funds, and difficulty with the cost￾sharing obligations required of many partners. Many such concerns dated from earlier days, and on balance the responses bespoke a group of partners that appreciates Avert's technical and material inputs, and sees their partnership as having a bright future for expansion and innovation. 6 FINAL REPORT II.B. FIELDWORK COVERAGE The evaluation team visited Avert Society NGO partners and other stakeholders in all seven priority districts, as well as other districts where “cross-cutting” activities are taking place (see below for a discussion of Avert's targeting strategy). To broaden its coverage it frequently divided itself into two-person sub-teams, with the result that it was able to visit almost 40 of the 54 NGOs and other stakeholders with which Avert had funding relationships as of November 2005. Annex A lists all of the NGO partners and others with whom the team had contact. The team's detailed field visit schedule appears as Annex C. On all visits to the field, teams and sub-teams were accompanied by a member of the Avert Society technical staff. Far from biasing the comments of NGO partners, this in fact enabled the team to have broader discussions than would otherwise have been possible, in which sub-grantee concerns were frankly aired, solutions to problems discussed, and ideas shared. It also gave the team insights into organizational management issues that are discussed later in this report. In addition to meeting with partners, the team spoke with MSACS and MDACS leadership, with Johns Hopkins (JHU) and Hindustan Latex Family Planning Promotion Trust (HLFPPT), both recipients of support under the technical component of USAID funding for Avert, and with other stakeholders, such as Family Health International (FHI) and Operation Lighthouse/PSI. It spent many hours in meetings and follow-up discussions with Avert technical and financial management staff, singly and in small groups. EVALUATION OF THE AVERT SOCIETY PROJECT 7 III. FINDINGS AND RECOMMENDATIONS III.A. AVERT SOCIETY PROGRAM STRATEGY The mission of the Avert Society is to increase the use of effective, sustainable methods for reducing transmission, and mitigating the impact, of sexually transmitted infections (STIs), HIV and related infectious diseases in the State of Maharashtra. As noted, the project is implemented through partnerships with NGOs and other stakeholders. Avert's Governing Board, chaired by the Secretary of Health of Maharashtra with a representative of USAID/India serving as Vice Chairperson, provides policy guidance and approves major funding and strategy decisions. The Avert Society Project Director, Associate Project Director and technical and administrative staff are responsible for project implementation. The Avert Strategic Plan has five components, divided into two categories: 1. Primary areas of activity: • prevention of transmission of HIV through targeted interventions (TI) with groups at high risk of transmission – commercial sex workers (CSW), truckers, migrants, slum dwellers and others – as well as workplace interventions • provision of comprehensive care and support (C&S) services for people infected and affected by HIV 2. Cross-cutting activities that support the primary areas: • capacity building of NGOs and other stakeholders to ensure maximum impact of their interventions • increasing the availability and use of research data to inform policies and improve programs • communication – involving mass media and other tools, including social marketing, to create a supportive context for HIV prevention and care for PLHAs, and developing NGO communication capacity and skills 8 FINAL REPORT Targeted interventions with groups at high risk of HIV transmission, except for the new (and highly innovative) category of workplace interventions (WPI), are limited to seven districts of Maharashtra – Aurangabad, Mumbai, Nagpur, Sangli, Satara, Solapur and Thane – chosen for their high HIV prevalence.3 This targeting of specific districts was designed to avoid overlap with MSACS, which supports TI activities of a similar nature in other districts. C&S services and WPI, generally not offered by other agencies, are implemented by Avert under a “state-level mandate” that enables them to take place wherever promising opportunities present themselves. Similarly, Avert's “cross-cutting” support for capacity building, research and communication, as well as condom promotion and social marketing, are implemented on a statewide basis. Can the strategy meet the challenge? Questions have been raised, by USAID/Delhi and others, as to whether Avert's complex formulation of focusing TI initiatives on seven “target districts,” while at the same time fulfilling a “state-level mandate” for other services and cross-cutting initiatives, is unnecessarily confusing and potentially counter￾productive. In fact, the evaluation team found that this strategy, if properly implemented, places the agency in an excellent position to provide a comprehensive response to the HIV epidemic. It gives them a laboratory (the seven target districts) for in-depth testing of interventions with a range of high-risk groups. At the same time it provides a chance to take a leadership role statewide, one that does not duplicate the work of others, in developing models in several key areas that will complement and reinforce Avert-supported targeted interventions. These include new approaches to care and support, WPI, and capacity building that can benefit everyone. However, the evaluation team does worry that, in its attempt to meet challenging implementation goals, Avert could overreach and risk compromising program quality. As previously noted, it has set itself an ambitious target of signing memoranda of approval (MOU) with close to 100 NGO partners by the end of March 2006. This will still leave unanswered numerous other requests for assistance that Avert has received over time from NGOs across the state. In other words, there is great potential for continued expansion. Nevertheless, before considering expanding its partner list further, the team urges Avert to take the time needed to fully assess the impact of the performance of existing partners, and ensure the quality of its support thereof. This evaluation report reviews Avert's work with TI, C&S, communication, capacity building and research initiatives, and makes a number of recommendations as to how they can be strengthened and expanded. Establishing a moratorium of at least six months on approving new partnerships will give the staff the chance to implement those recommendations it decides to adopt, put new systems in place, and strengthen its technical inputs. It will give the project time to determine which activity areas, whether state-level or in target districts, should receive highest priority, and will enable recommended new staff positions to be integrated into operations without undue pressure. Stakeholder synergy. A second concern with respect to Avert's program strategy has to do with the extent to which it meshes with the work of the other two leading Societies supporting HIV prevention efforts in Maharashtra. The team found that MDACS and MSACS are well aware of the need to ensure that their strategies are complementary to 3 According to MSACS data of December 2004, Mumbai has the highest number of AIDS cases and AIDS deaths, followed by Sangli, Kolhapur, Thane, Pune, Satara, Latur, Nagpur, Raigarh and Ahmednagar, in that order. EVALUATION OF THE AVERT SOCIETY PROJECT 9 those of Avert. It found that the other Societies welcome Avert's initiative in implementing activities, such as WPI and C&S services, that are not part of their strategies. It was encouraged by the formal and informal mechanisms that exist to avoid duplication and confusion of roles. However, this is an issue that needs continuous monitoring. In the case of MDACS, although it is formally responsible for HIV prevention programming in Mumbai, the district is also a priority target for Avert and the site of a number of its TI initiatives. MDACS leadership welcomes this partnership, and asserted to the team that the two agencies are “very much in touch, every day, all the time.” MDACS looks to Avert for training inputs for staff of the twenty or more NGOs it supports for TIs in Mumbai, and the two agencies have worked closely together on World AIDS Day promotions and other public efforts. In the case of MSACS, the potential for duplication of effort is greater. As described, the Avert Society and MSACS have delineated specific districts in which they each support targeted interventions among high-risk groups: Avert funds NGOs implementing TIs in seven districts, while MSACS supports TIs in the other districts of Maharashtra, presently working through 38 separate NGOs. MSACS, in collaboration with the State Department of Health Services (DHS), is also tasked with assuring a dependable supply of condoms for interventions statewide. Avert's initiatives in WPI, C&S services and capacity building are able to go forward under its state-level mandate, including in “MSACS districts,” because these are not areas in which MSACS is active. The MSACS director emphasized to the evaluation team the benefits of “common sharing between the agencies of their skills and resources”. He stressed the importance of continually guarding against unnecessary duplication of effort, but emphasized that since "we want super saturation [of services in the state], overlapping is not always bad." Governing Board. Both MDACS and MSACS are represented on the Avert GB. This provides a logical, regular forum in which to review collaborative efforts, and ensure that the resources of each are being used to maximum advantage. The GB's role, and how it can be strengthened to the advantage of the Avert Society program, is discussed later in this report. It should also be said that the location of the Avert Society office within the same compound as MDACS and MSACS has proven strategically critical for an ongoing collaborative relationship and a culture of sharing among the three organizations. As an official from one of the Societies put it, “We don’t knock on each others' doors …we just walk in.” RECOMMENDATIONS: • After meeting March 2006 goal for new NGO partnerships, institute a six-month moratorium on approving new partnerships (except for exceptionally innovative opportunities) to enable those already in the fold to be fully established and to take steps needed to further strengthen technical support. Use the forum of regular Governing Board meetings to review and monitor efforts on the part of MDACS, MSACS and Avert to avoid duplication of effort among the three Societies and thus maximize their resources in combating HIV statewide 10 FINAL REPORT III.B. HIV AND AIDS PREVENTION THROUGH TARGETED INTERVENTIONS: NGO Partnerships. The largest component of the Avert program is its effort to prevent transmission of HIV through targeted interventions with high-risk groups in the project's seven focus districts. To accomplish this, Avert has established partnerships with NGOs that are experienced in the field, and with those that are new to the work but show promise. It has also established partnerships with corporate entities, through its WPI initiative (see below), reaching out with prevention messages and services to workers and labor unions. Partners are selected from a long list of applicants, according to a rigorous set of criteria that measure an NGO’s competence, commitment and potential. In visits to almost 40 such partners in the seven districts, the evaluation team held wide-ranging discussions with NGO officials, outreach workers (ORWs), peer educators and people in the community. It concluded that the NGO selection process has, by and large, resulted in providing Avert with a creative, committed roster of partners, and came away impressed that, through these partners, the Avert program has achieved significant, positive visibility. NGO and other partners see Avert support as an opportunity to complement their existing programs. The team found them determined, after the expiry of support, to maintaining capacity to provide prevention services. This is particularly true for corporate sector partners. A senior official of Bharat Petroleum, a corporate recipient of Avert technical and financial assistance, said that “this partnership with AVERT has been beneficial … We are exposed to many intricate and sensitive issues that we [earlier] never appreciated. The program has helped to sensitize senior officials in the company. Frankly speaking, we don’t need money from Avert [as much as] we need an ongoing technical assistance and guidance that we have never received before”. High-risk target groups. TI activities supported by Avert through its partners are directed to populations likely to engage in high-risk behaviors. As is traditionally the case with prevention programs, targeted groups include commercial sex workers (CSW), men who have sex with men (MSM), truckers, and migrant workers from slum communities. Avert has also exhibited creativity and flexibility by reaching out to an innovative range of “hard-to-reach” groups that include people at workplaces in the unorganized sector, the slum community in general, transgenders, hidden sex workers, and prisoners. As noted, Avert has done a good job of NGO selection, one that has moved it in the right direction in saturating high risk groups in the seven target districts. However, one area of weakness is that mapping data is seldom used effectively to plan interventions, and the data, as collected, is itself of questionable quality. There is a clear need to validate mapping data and also train the NGO partners to use the mapping exercise effectively for program planning. Conceptually speaking, all components of Avert TI initiatives are mutually reinforcing and complementary. However, as will be discussed later, there are operational difficulties and challenges. For example, reliable referral linkages and follow-up mechanisms for STI treatment and control are lacking; condom promotion is hampered by irregular supply and poor quality; BCC has been largely in the form of information giving and isolated “mega-events.” EVALUATION OF THE AVERT SOCIETY PROJECT 11 An emphasis on interventions exclusively with high-risk groups has its limitations and needs to be continually reviewed, particularly in a program that attempts to reach a wider population. Such activities have a better chance of long-term impact, and their components are made mutually reinforcing and complementary, if their focus is on particular behavior rather than on particular groups of people. An all-encompassing “high-risk” approach is likely to objectify and label people, without taking note of covert behaviors and the situations that might be responsible for risk. An approach exclusively focused on high-risk groups is also likely to stigmatize and isolate people within the wider community. For this reason, Avert support for PLHA associations and other groups whose membership cuts across social class lines within the community is a key element of its strategy (see more on this below.) Front line workers. TI leadership rests with outreach workers, full-time NGO staff with Masters in Social Work (MSW) degrees, usually recruited from outside the area in which they work. Activities for which ORWs are responsible include: • recruitment and training of peer educators in the target community • “one-to-one” and “one-to-many” communications • promotion of condoms, using free and social marketing channels • promotion of appropriate STI treatment seeking • advocacy in the community to create an enabling environment ORWs of the NGOs visited by the evaluation team had all benefited from Avert training courses on the basic facts on HIV/AIDS, sex and sexuality (offered by the Sosva Training and Promotion Institute, STAPI), and from training of trainers (TOT) of peer educators provided by the BIRDS program. ORWs were appreciative of the quality of the training received, but with some reservations. For example, it was evident from our discussions with ORWs and NGO officials that there was an urgent need for sensitization on gender issues and on combating stigma related to PLHAs. Many expressed the need for hands-on training in dealing with other, related issues, such as alcoholism and domestic violence, and for linkages with the legal system and rehabilitation centers for battered women and alcoholics. During its visits to NGOs the team also observed several instances of a high level of turnover of program staff, particularly ORWs. This has serious implications for the continuity of the program and should be addressed jointly by Avert and NGOs. The evaluation team felt that the Avert Society’s focus on recruitment and development of cadres of peer educators is one of the great strengths of its targeted intervention program. Most NGOs have recruited large numbers of CSWs or other members of target populations to themselves reach out to and educate their peers in the community. They are volunteer workers, whose training and oversight are the responsibility of NGO personnel, primarily their ORWs who, as described above, have received TOT through Avert. The evaluation team met many energized, articulate and committed peer educators in the field. It was clear to the team that a key to ensuring long-term impact and sustainability 12 FINAL REPORT of the program must be to retain the interest and motivation of these workers. TI program strategy should be to promote and sustain this unique cadre of community leaders. Communication. ORWs and peer educators use a variety of IEC materials – flip charts, brochures and pamphlets – as aids in one-to-one and one-to-many communications as part of targeted interventions. Often, however, the team heard that non-availability of materials in local languages presented problems for outreach. Moreover, communication activities in the field do not seem to follow any particular strategy. Identification of individuals and groups is opportunistic, and communication tends to begin and end with the provision of information. ORWs and peer educators provide crucial and, in many communities, the only visible linkage between the community and information and services at large. In this sense, they are the best potential agents for communications and behavior change. Some, especially peer educators, are very creative. They have strong roots in the community and an intimate understanding of their peers’ information needs. Given this resource, it is imperative to implement communication activities within a broad BBC strategy, one that uses existing community resources and channels rather than carrying out stand-alone activities. More discussion of the need for a holistic, multi-layered approach to BCC strategy within the Avert Society project appears later in this report. Condom availability. An important feature of the community-based TI program is making condoms accessible to the target population. This is done through enrolling support from “non-traditional” outlets (as opposed to regular merchant shops and medical outlets) and peer educators. The team observed youth clubs, petty business outlets and influential individuals who stored condoms for free distribution. NGO records generally revealed high uptake of free condoms, with some NGOs using their own local networks to meet a large demand and maintain regular condom supply. The team even had the opportunity to witness “live” demand for condoms during a meeting in Aurangabad, when an NGO project coordinator received a text-messaged request for his next stock of free condoms on his cell-phone! Many NGOs, however, complained of irregularity in condom supply. The team’s impression was that there is a large unmet demand for condoms in the community, and a need for a strategic intervention to maintain regular supply of free condoms. Most NGOs were not comfortable with the idea of condom social marketing and were clearly not keen on selling condoms. On the other hand, there was a significant minority of NGOs that supported the idea of CSM, and felt that socially marketed condoms could eventually meet demand. Further discussion of how CSM could be gradually introduced appears later in the report. STI referral and treatment. As part of targeted interventions, ORWs and peer educators spend a considerable amount of time identifying and referring “suspect” STI cases from their targeted population to appropriate service outlets. By and large, identification of STI cases in the field is ad hoc and opportunistic. Most ORWs lack confidence in their capacity to syndromically assess STIs. Many NGOs working on prevention do not have consistent referral systems in place. Often they identify local doctors to whom they refer STI cases, while others refer their cases to nearby government hospitals and clinics, with little or no follow-up. Issues concerning the STI treatment of CSWs and their clients were especially problematic, given the fact that EVALUATION OF THE AVERT SOCIETY PROJECT 13 CSWs are unorganized and live in an extremely hostile environment. Several sex workers talked to the team about the violence they face at the hands of police and local “goons.” There is much that can be done to improve systems of STI diagnosis and referrals in connection with TIs, as well as to clearly establish, in the minds of persons contacted, the links between STI and HIV. A first step should be to undertake a rapid assessment of STI treatment seeking behavior, including how target populations recognize signs and symptoms of STIs, at what stage of the disease progression they seek services, and from whom. What we already know from a wide range of literature on the issue is that informal sector and indigenous health providers play a significant role in the provision of services. Avert would do well to develop programs for involving indigenous health practitioners in diagnosis and treatment of STI. Further recommendations for enhancing STI referrals and services are included in the C&S section of this report. Community awareness. Activities that most NGOs cherish and look forward to are community-based “mega-events” that they organize around the time of World AIDS Day. (This is actually a misnomer, since celebrations often span up to a week, featuring large group activities and public mobilization.) The Avert Society evaluation coincided with World AIDS “Week” and the team had an opportunity to witness energy and innovation in Avert operations areas. A typical event brought together CSWs and people living with HIV and AIDS to manage public information booths and exhibition stalls, distribute condoms, and talk with the general public about safe sex and the importance of displaying sensitivity towards PLHAs. It is important that such events be continued, with a view to giving a “face” to HIV, to help dispel the myth that HIV is a disease of “others.” At the same time, however, there is a need to maintain continuity and to follow up these large events to ensure that preventive and C&S support services are reaching those who need them most. RECOMMENDATIONS: • Avert is urged to work with partner NGOs to integrate TIs for HIV prevention with ongoing community-based activities, rather than creating new “vertical” programs. While doing so the focus should be on “risky context and behavior” rather than on “people practicing risky behavior.” • Working towards a true “continuum of care,” NGOs implementing TIs need to establish more effective linkages with C&S services and testing facilities, starting with a rapid assessment of STI treatment seeking behaviors. Special emphasis needs to be placed on follow-up of STI and VCT referrals. • Sustainability of targeted interventions depends on the skills and motivation of peer educators. TI program strategy must sustain this unique cadre of frontline workers, including building their capacity to access local resources. • Every effort must be made to ensure greater gender sensitivity and sensitivity to PLHAs at all levels, starting with greater involvement of PLHAs in project activities. • Avert is to be commended for its strategic alliances with corporations and industry groups through its WPI initiative, and is urged to expand this initiative statewide, with emphasis on the unorganized labor sector. 14 FINAL REPORT • While TIs have done effective targeting, there is a need to reach out to other hard￾to-reach, high-risk groups, such as clients of CSWs, families of truckers, bisexual men and youth, especially those living in hostels and dorms. • A dependable supply of condoms is crucial to continued credibility of TIs, given that there is a large unmet demand. III.C. WORKPLACE INTERVENTIONS Mention has been made of Avert's targeting of HIV prevention messages to workers in corporate entities, formal and informal industry groups and labor unions. Through this unique and innovative effort, Avert has been able to reach out to both organized and unorganized labor sectors. Impact on the former is exemplified in the earlier quote from an official of Bharat Petroleum. Large corporations have often lacked conviction about HIV prevention and the value of C&S. Many have fixed procedures regarding medical check-ups and reimbursements, and feel HIV can be handled like any other disease. Avert's WPI initiative has been able to disabuse companies of this attitude, and demonstrate the value to their business of providing appropriate HIV and STI information, counseling and treatment. This effort is strongly supported by Avert's excellent manual on “HIV/AIDS Management at the Workplace,” which provides step￾by-step procedures for developing workplace interventions. In the unorganized sector, Avert works closely with Naka workers (daily-wage male and female workers) in Mumbai, one of the largest groupings of migrant laborers, and with loose organizations of CSWs. The team spent considerable time with ORWs and peer educators working among these groups, in Mumbai and elsewhere, and was convinced of the importance of expanding WPI outreach to them as well as to companies and organized industry groups. From discussions with NGOs and other stakeholders active in both formal and informal labor sectors, the evaluation team also came to believe that WPI initiatives can be even more successful if linked with counseling for alcoholism. RECOMMENDATIONS: • Avert has the WPI field “all to itself,” and should make the most of the opportunity, seeking to expand interventions in both unorganized and organized sectors. (We would even suggest that this area of growth not be subject to the moratorium on new partnerships previously recommended.) • Build partnership with NGOs that are working on alcohol interventions among migrant workers and set up programs combining alcohol and HIV prevention. • Initiate advocacy programs with companies and industry associations on assuming responsibility for HIV+ employees and their families. This may include seminars and discussions in the media on the impact of HIV on productivity and responsibilities of corporate houses in mitigating the impact. • Widely disseminate the Avert WPI manual and document its utilization. • Document success stories as well as highlight challenges on the part of corporate sectors to respond to HIV/AIDS needs of workers. EVALUATION OF THE AVERT SOCIETY PROJECT 15 III.D. CARE AND SUPPORT SERVICES Several types of care and support services are supported by the Avert Society through partner NGOs, their common objective being to meet the needs of PLHAs for an effective continuum of care, the elements of which are outlined in the accompanying box. Although, as detailed below, there is room to strengthen different aspects of C&S services offered by Avert partner NGOs, the team felt that Avert's C&S strategy is sound, and that the range of activities that it supports are providing important models for further development and expansion. 1. Community Care Centers (CCC) CCCs provide comprehensive services to the infected through fixed, in-patient sites set up in hospitals or other health institutions. All follow NACO guidelines for CCC operations. Patients are admitted for up to two weeks of treatment for opportunistic infections (OIs), and psychological and nutritional care, with the expectation that they and their families will receive on-going support once they are released back to their communities The evaluation team visited four of six CCCs supported by Avert. Physical infrastructure. CCCs visited have adequate infrastructure for admission of patients. Lata Mangeshkar Hospital in Nagpur, Bel-Air Hospital in Panchgani and the John Paul Trust in Pune have separate wards for their CCCs, with about ten beds dedicated specifically to admission of HIV+ patients. By contrast, in the CCC in Solapur beds are scattered throughout the hospital. The team feels that designating wards specifically for HIV+ patients tends to increase stigma and discrimination, and does not contribute to mainstreaming HIV. Designating "Avert wards" or "HIV wards" should be discouraged, with CCC beds dispersed throughout their institutions as far as practicable. Medical Personnel. CCCs have medical officers with varying qualifications in charge. One has a gynecologist and an anesthetist handling HIV patients. A second has a surgeon in charge, a third an ayurvedic physician. With a few exceptions, none have undergone formal training in HIV management. As a result, diagnosis and treatment of OIs, as well as referral for ART medication, vary widely, and there tends to be an over￾diagnosis of OI. In one of the CCCs visited, an average of four OI were diagnosed for each patient admitted! CARE AND SUPPORT – NEEDS OF PLHA • Medical - Opportunistic infections - Identification - Treatment - Prophylaxis - Nutrition and Hygiene - ? ART • Psychological - Reaction to the disease - Preexisting Psychological problems - Addictions - End of life issues • Social - Reduction of stigma - Employment - Income generation - OVC • Legal - Property - HR / gender issues • Spiritual - Differentiate Religion, rituals and spirituality 16 FINAL REPORT At least one medical officer in each CCC should have basic qualifications in allopathic medicine recognized by the Medical Council of India, since only MCI registered practitioners can legally prescribe allopathic drugs. Training for medical officers should familiarize them with all aspects of CCC services (see below) and NACO CCC guidelines, with salaries made commensurate with qualifications and experience. Medical officers should be encouraged, and assisted, to attend at least one national conference each year. With respect to nurses, most CCCs are staffed by RNs, and nursing care in general appears to be good. However, many nurses are not aware of post-exposure prophylaxis or procedures to be followed after occupational exposure to HIV. Nurses have a major role to play in CCCs as informal counselors to PLHAs and their families. This places a premium on knowing the local language, which the team observed was not always the case. Nurse training should include sessions on informal counseling and post-exposure prophylaxis, with nurses encouraged as much as possible to learn local languages. Finally, some CCCs have made efforts to recruit PLHAs as support staff. This is absolutely a practice that needs to be encouraged in all centers supported by Avert, as a step towards making inroads on the influence of stigma. Laboratory Services. These too vary with different CCCs. For example, Lata Mangeshkar Hospital is attached to a medical college, with facilities for diagnosis of opportunistic infections. Other sites do basic hematology, biochemistry and sputum examination, but microbiological services are non-existent. Services at the John Paul Trust are still being established. Laboratories are managed by technicians with varying levels of training, with little quality control. Laboratories need to be upgraded in terms of instruments and reagents. Avert should explore the possibility of linking with local microbiologists for supervision of laboratory services. A course for lab technicians for diagnosis of OIs is urgently needed. Standardized training of staff and improvement of laboratory services will reduce over-diagnosis of OIs. Drugs for opportunistic infections. Drugs are necessary for treatment of OIs as well as for prophylaxis. At present, CCCs either provide drugs themselves or patients buy their own drugs. This is a major problem for the institutions, one that could be eased if Avert can prevail upon the Department of Health Services to provide drugs to these CCCs, which are, after all, offering a unique and sorely needed service. Centers can be linked to the nearest public health center so that the drugs can be accessed without difficulty. A Government Order to this effect would help Avert and its CCCs secure a dependable supply of drugs for OIs. Nutrition. Avert support of CCCs provides for nutrition services and supplementation. At Bel Air Hospital, patients receive food from a common, hygienically managed kitchen. In other centers, the supply of food is outsourced, and its nutritional value is not clear. Also, since patients usually have family members staying with them, food tends to get shared, rather than used for the patient's nutritional benefit alone. Avert needs to develop a consistent plan and guidelines for nutritional support of CCC patients, and contract with a dietician to provide nutritional training to all CCCs. Provision of food for relatives should also be considered. Training of family members. A CCC must prepare the family to take care of their infected family member, especially once he/she leaves the center. All Avert-supported CCCs insist that a relative accompany the patient in the facility. However, except for Bel EVALUATION OF THE AVERT SOCIETY PROJECT 17 Air Hospital, there is no apparent training provided to the relatives on care and support of the infected person. With Avert's assistance, CCCs must make an effort to educate the family in C&S. This can be strengthened if it is introduced in pre-program training for all categories of Avert partners. Referrals and Linkages. A fully functional CCC needs to have both forward and backward referral linkages. If the center cannot handle a patient, there should be a tertiary care center where he/she can be referred. Most CCCs do not have an orderly forward referral system. Patients are referred to anti-retroviral therapy (ART) services in an ad hoc manner. CCCs also have poor linkages with PLHA networks and other Avert partners. This may be due to the fear of losing patients to other agencies. As one NGO contact said to the evaluation team, “once we refer the patient then we lose the patient, and only when they become hopelessly sick do we get them back.” The formation of NGO forums in focus districts, such as the one recently launched among Avert Society partner NGOs in Nagpur, will improve referral networks and should be encouraged. Avert should also negotiate with other stakeholders, such as MSACS and the directorate of health, to build forward referral linkages to ART services. Community Outreach. By definition, CCCs are designed to develop and maintain outreach to, and involvement of, the local communities in which they are located and from where their patients come. This is so that they can maintain contact with and monitor their patients after release, as well as stay in touch with their families. Outreach is also designed to gradually make the community better informed about HIV and AIDS, and thus work to diminish and eliminate the attached stigma. The evaluation team found that CCC outreach operations are of mixed quality, and none are mature. Patients are often lost to follow-up because of inaccurate addresses, lack of trained ORWs, or family or community opposition to outreach visits. This is an element of CCC services that needs continued analysis and strengthening. The CCC as model. Community care centers are a good model for providing care and support services, though one that can be significantly strengthened. Patients who need admission and treatment for OI get them. At the same time, the families learn about HIV as well as ways to cope with the disease and the sick person. All CCCs provide terminal care for patients as needed. The home-based care component of the CCC can be improved. Each of the centers should have a minimum of three staff persons, so that personnel are available to make home visits and strengthen the “community” aspect of the service. The basic qualification should be that of health visitor, with at least three months training in HIV management. Provision must be made for travel, using local public transport. Provision of a vehicle for the program should be strongly discouraged. Referral linkages need to be built between anti-retrovirals (ARV) centers and the CCC. The CCC medical officers and ARV centers should meet at least once a month, to ensure coordination between them. Avert should coordinate with MSACS and the state health department to be sure this coordination occurs. At present, there are no linkages between voluntary counseling and testing centers (VCTC) and the CCC. One way this gap can be bridged is to have a specified VCTC linked to a CCC. Similarly, VCTCs should have a designated center where the patients can be referred for OI management. Before a VCT center is sanctioned, Avert should 18 FINAL REPORT conduct a survey among government and private practitioners on the utility and location of the VCTC in the particular area. 2. Drop-in centers (DICs) Avert supports five drop-in centers run by and for PLHA networks, and three DICs for sexual minorities (MSMs and transgenders). PLHA Drop-in centers. An encouraging development for the future is the decision on the part of many, usually young, people living with HIV and AIDS not to hide their status, but to form support groups for those willing to admit that they, too, are PLHAs. One tangible reflection of their decision is their establishment of drop-in centers for HIV+ people where, in the words of one DIC leader, "they can come, they can talk, they can cry," which in turn helps them take control of their lives. The DICs, usually staffed by PLHAs, attempt to normalize HIV in the community by arranging public awareness programs, and often are able to get free space from their municipalities. The DIC in Sangli has arranged marriages between HIV infected members. Another center has arranged “nature trips” for HIV+ and non-infected persons as a way of combating stigma, although the team found this an ineffective approach that should be discouraged. What is especially encouraging, and what nurtures the hope that establishment and nurturing of drop-in centers can help reduce the fear that surrounds HIV, is that such groups tend to span social classes. At Avert-supported DICs the evaluation team met not only farmers and laborers but housewives, small business people, educated folk from different levels of society. Supporting groups of HIV+ people willing to tell their stories, and centers where they can gather, seems an especially hopeful investment. DICs typically have a gathering area, a counseling room, and space for reading matter and informational materials. Issues facing DICs include the need for referral linkages with the medical establishment, and with treatment and C&S facilities. Some have linked up with private practitioners, others are trying to do so with VCTCs. All DICs expressed the importance of having medical consultants on site on a regular basis. At the Aamhich Aamache DIC in Sangli, for example, attendance increases markedly on days when a medical consultant is present. DICs are typically open from 9 or 10 a.m. to 6 p.m., which seemed to the team too limited, given their purpose. The centers are also would like PLHA members who need them to be able to access food rations. This could increase attendance and would be, as one DIC organizer said, “the least that we can give to the clients.” To strengthen the DIC model, as it matures and expands throughout the state, centers need to become even more user-friendly. Hours should be adjusted so that more people can access services. Avert should make certain that DICs build linkages with VCT and C&S facilities, and make provision for a part-time medical officer. Community awareness programs should expand their reach. On the other hand, provision of food rations should not be encouraged, as it will encourage dependence. Instead, income generation schemes should be developed. The centers could, for example, encourage animal husbandry as a way of increasing economic independence. Finally, a clear-cut distinction has to be made between the functions of DICs and home-based care. Drop-in centers for sexual minorities. All such centers provide space for sexual minorities to meet, and are open mainly in the evenings, so as to be accessible to their EVALUATION OF THE AVERT SOCIETY PROJECT 19 clientele. Staff are very committed, and provide STI and HIV prevention education to DIC users. They work hard to identify new clients and actively encourage condom usage. When clients are diagnosed as HIV+, they tend to discontinue coming to the centers. One of the workers told the team that ‘people cannot live with two stigmas’, and as a result they drop out, sometimes only returning during the last stages of AIDS. All centers have identified medical practitioners empathetic to their cause who provide services to their clients. 3. Home based care (HBC) Only one HBC initiative, ASTHA in Sangli, has so far received Avert support. There is a high level of commitment and motivation among the staff. But they need proper guidance and training, and the roles and duties of the members of the team must be clearly delineated. 4. Counseling All C&S services (CCCs, DICs, etc.) have trained counselors, most with MSW degrees. There is good gender balance among the counselors, with women usually having the choice of being counseled by women. Most facilities have set aside adequate space and privacy for this function, which includes pre-test, post-test and crisis counseling. However, the evaluation team observed that counselors are more comfortable with providing information than they are with problem solving. Interacting with the counselors indicated that they are not adequately equipped to identify psychological problems, and counselors expressed a uniform desire to receive more training and help in dealing with alcoholism. If they identify a counseling issue that they cannot handle, there is no mechanism to refer the patient or obtain a second opinion. In one of the centers, the director said that in those cases “we all put our heads together and try to find a solution.” The team feels there is an urgent need to appoint a counseling officer within the Care and Support unit of the Avert Society. The counseling officer would be available to all C&S facilities to address their counseling needs and issues. 5. STI services All activities supported by Avert have an STI component. In TIs for high-risk groups, ORWs attempt to identify individuals requiring STI counseling and treatment. Communication materials and the level of knowledge about STIs among ORWs are generally adequate. If clients have symptoms of STI, the attempt is made to refer them to medical practitioners. This, however, is a weak point of the program, as has been suggested elsewhere. Some referrals are to public hospitals, others to local practitioners. These practitioners vary in their qualifications, including some who can be categorized as “quacks.” There is little evidence that practitioners prescribe standard treatment protocols, and no effort is made to link STI with HIV. Few ORWs are trained to refer STI patients for HIV testing, nor do they try to identify patients’ partners and refer them to STI services. Finally, there is no information about follow-up of patients after they have been referred for STI services. 20 FINAL REPORT ORW training needs to lay more emphasis on total care (diagnosis, referral and follow￾up). Emphasis should be laid on the syndromic approach to STI diagnosis and treatment. Avert should organize trainings for all of the medical practitioners to whom STI patients are referred by ORWs, to ensure better treatment of STIs. 6. Centers of Excellence Avert has a unique opportunity to develop some of its care and support services into centers of excellence, to be used for training and to highlight Avert-supported activities. The Yerala Society in Sangli is one such possibility. It has a slum intervention program as well as a combination CCC and VCTC supported by Avert. It could showcase NGO cooperation and demonstrate a comprehensive community model. A second candidate is the Dr. Babasaheb Ambedkar Vaidyakiya Pratishthan, centered in the Dr. Hedgewar Hospital in Aurangabad. It has a large TI program in slum areas, links with other NGOs, an excellent training facility, and extensive technical resources. A third potential center of excellence is the Bel Air Hospital in Satara. It has the physical infrastructure for conducting training programs, and the quality of its medical care and counseling is of a high standard. The evaluation team urges consideration of the concept of developing one or more of these (or other) programs as centers of excellence, for the active, evolving demonstration of various approaches to care and support services. RECOMMENDATIONS: The Avert Society's care and support program is a work in progress, with great potential to be a leader in the field. To summarize recommendations contained in the previous paragraphs: • The Avert community care center is a model for care and support services worthy of being strengthened and replicated. • CCCs should make every effort to get close to, and reduce stigma in, communities they serve, by not isolating in-patients in “HIV wards” and by strengthening community outreach and training of family members. • Selection and training of CCC medical officers and nurses should be strengthened, emphasizing counseling skills and post-exposure prophylaxis. Training PLHAs as CCC support staff is highly desirable. • Issues regarding laboratory standards and training, improved access to drugs for OIs, and nutritional needs of CCC patients need urgent attention. • Referral linkages in all aspects of C&S services, especially for STI diagnosis and treatment, need urgent strengthening, starting with broadened training of ORWs. Another important mechanism is development of NGO forums in focus districts. • Drop-in centers, especially for PLHAs, have a critical role to play in supporting the afflicted, generating community awareness, and combating stigma. It is a model that should be expanded wherever possible. EVALUATION OF THE AVERT SOCIETY PROJECT 21 • The counseling component of C&S services needs significant strengthening, starting with appointment of a counseling officer in the Avert C&S unit. • Development of C&S centers of excellence within existing Avert partners should be pursued. III.E. COMMUNICATION Background. From the outset, the Avert Society project placed significant emphasis on the role played by communication, as a cross-cutting activity that would support and enhance its primary activity areas of targeted interventions and care and support services. It is of note, for example, that Avert provided for a communication specialist on its technical staff, indicating its intention that the position serve more than a standard “information, education and communication (IEC)” function. Major communication work of Avert to date has centered around activities such as World AIDS Day and World Health Day, and development of products such as posters, pamphlets and booklets. Materials are also produced at the NGO level, although apparently not always with technical assistance from Avert. In an effort to help NGOs with their need for communication tools, an India-wide search for appropriate materials was fast tracked, and some replicated for use. A communication strategy first developed in 2002 was further refined in 2004 in a process led by Johns Hopkins University. Indeed, the ongoing involvement of JHU, on contract under the technical component of USAID funding of Avert, is another indication of the desire for communication to play a key role in Avert. The strategy document provided insight on channels that were appropriate for disseminating messages, but paid little attention to message content. It dealt with communication at community, district and state levels, but contained no guidance on development of messages for particular target audiences. Current program issues. The evaluation team found the Avert communication program lacking in a holistic concept of its role, and insufficiently focused on behavior change. Communication skill training in the training programs offered by Avert seems routine and not commensurate with any specific requirements. Medical fraternities have been left out of any sensitization program. There has been no specialized training on the communication process and its application to the work of counselors, ORWs and peer educators, which has hampered them in working to their full potential. Sensitization of Avert staff itself on the role of communication in project implementation and program management has been given little time. Most, if not all, communication materials produced by Avert have been oriented to providing information on HIV and AIDS. Even in one-to-one communication the emphasis has been on knowledge dissemination. There has been little emphasis on benefits that the audience will obtain if they act on that knowledge. Failure to understand a particular audience’s concerns and to design messages relevant to them has been a major shortcoming. Communication as a process needs strong linkages between channel and content. In the case of Avert, the process has primarily been media product￾driven and ad hoc. A major advocacy thrust is also conspicuous by its absence from the Avert communication approach. Some advocacy work is underway through the workplace 22 FINAL REPORT intervention initiative. But focus on strategic advocacy intervention does not carry through to the ground level. Reorienting focus to behavior change. It is important for Avert, its partners and stakeholders to realize that this is a behavior change program, where the context is HIV and AIDS. In the absence of this thinking there is a tendency to compartmentalize the program into components such as “targeted interventions” and “care and support.” It will augur well if behavior change communication becomes the binding factor. This will help in giving all actions a common goal. For example, whether one is setting up a DIC or VCTC, designing a training program or creating support material for one-to-one communication, the question uppermost in one's mind should be whether this action will help in changing behavior in such a way as to prevent HIV transmission. Personnel. Avert requires a communication specialist with proven health and/or social communication expertise, skills not presently on staff. This should be someone who can move the organization toward an integrated, BCC-centered approach to its work, and reorient the staff to the cross-cutting role of communication in the overall Avert Society program. The team also feels there would be great benefit in recruiting two or three communication coordinators, supervised by the Mumbai office, to be located at district or regional levels. Responsibilities of these coordinators would include contributing to content design by providing insights to audiences’ mindset, conducting communication process orientations for ORWs and peer educators, and assessing correct use of communication materials. Other activities would include organizing community meetings and other events, and documenting cases. Coordinators would also be an integral part of capacity-building exercises. Role of JHU. To date, Avert has not taken full advantage of the resource it has at its disposal in its contract with JHU for ongoing technical assistance for communication activities. But collaboration between the two offices is good, plans are in place on which to build, and there is every reason to be confident that Avert can maximize the benefits of this association. A good place to start would be a joint review of Avert communication strategy, and agreement on steps to be taken to reorient it to a BCC mode. Avert and JHU should hold periodic workshops with Avert staff and NGO partners on such topics as “How behavior change communication works” and “Planning communication initiatives.” Avert will need to ensure that before such workshops there is a follow-up plan agreed by everybody, with timelines. Introduction of the ABC approach (Abstinence, Being faithful, and Correct and consistent condom use) to preventing HIV transmission has made content designing more complex. NGO partners need to understand the importance of feeding relevant data, both quantitative and qualitative, into the process of designing content and messages. There are also Maharashtran linguistic groups that need attention, such as Urdu, Telegu and Kannarese. Training. Another area that needs urgent review is the way that communication skills development and communication process orientation is integrated into Avert training programs. (See further discussion in following section on capacity building.) The audiences that Avert NGO partners are trying to reach vary widely, from CSWs to PLHAs to prisoners to organized and unorganized workers in various industries, and the environments in which communication must reach them differ widely. In a situation EVALUATION OF THE AVERT SOCIETY PROJECT 23 such as this, it is essential to be able to clearly identify the problem being addressed and the desired response, so that both messages and choice of media are sensitive and appropriate. Avert and JHU will need to review and modify training curricula to ensure that they are appropriate for building these skills. (Another technical assistance resource to be considered is Operation Lighthouse/PSI, which has developed very successful approaches to message development and dissemination, particularly with respect to interpersonal communication.) Advocacy. Whether in the workplace, in the mass media or at the level of the community, skills in advocacy are essential to addressing stigma and for general sensitization. Although the existing Avert communication strategy document does address this, little related work has been carried out as yet. Areas that need attention are: ƒ Advocacy in the workplace ƒ Advocacy in mass media ƒ On-ground advocacy with community, family, police, etc. ƒ Sensitization advocacy with the medical fraternity Advocacy strategies need to be developed so as to be initiated as a campaign rather than one-offs. JHU has initiated some work in this regard, providing a basis from which to generate more comprehensive action by Avert. RECOMMENDATIONS: • Be clear that Avert's communication efforts are part of a behavior change program, undertaken in the context of HIV and AIDS, rather than a program that simply provides information and IEC events and materials. • Bring a communication specialist (or specialists) with health and/or social communication expertise onto the Avert technical staff. Consider adding communication coordinators in the field to give added impetus to holistic, integrated approach to communication activities. • Maximize JHU technical resource, in refocusing/broadening communication strategy, placing new emphasis on message content, upgrading training curricula, expanding advocacy initiatives, staff recruitment, etc. This is a key resource, to be fully exploited. • With JHU, organize periodic workshops on key issues for Avert staff and partners. Possible topics: understanding BCC, the ABC approach, etc. III.F. CAPACITY BUILDING Building the long-term capacity of Maharashtran NGO partners to conduct HIV prevention and C&S activities beyond the duration of Avert support is a major cross￾cutting area of project activity. An indication of its importance is the fact that Avert has a senior capacity building specialist on its technical staff. 24 FINAL REPORT Training. Avert has engaged three reputable organizations to provide regular training courses for NGO partners. The Sosva Training and Promotion Institute (STAPI) in Pune provides a quarterly course for NGO staff, especially ORWs, on basic facts about HIV and AIDS. The Tata Institute of Social Sciences (TISS) holds training courses for counselors. BIRDS trains peer educators. Participants to whom the evaluation team spoke expressed appreciation for this training, but their evaluations were not very objective, since it was clear to the team that there are ways in which training offerings can be strengthened. For example, counseling training courses focus heavily on HIV counseling. They teach participants how to provide information to their clients, but they do not prepare them to identify or deal with psychiatric or other problems that may surface in the course of an interaction. Also, there is at present little or no follow- up or refresher training for counselors. As previously mentioned, the team feels the counseling component of Avert C&S services needs strengthening, starting with appointment of a counseling officer in the C&S unit. This should also include a review and upgrade of the TISS counseling curriculum. The same can be said for other training curricula, in the sense that the field is constantly changing and the relevant curricula need to adjust. As a start, the team urges Avert technical staff other than the capacity building specialist (who always attends) to sit in at all training courses, unannounced, so as to monitor them for content and consistency. Findings from research and NGO monthly technical reports (MTRs) should be incorporated into training programs. Trainers should also visit trainees in the field for follow-up instruction and advice. Looking more broadly, Avert needs to go beyond simply scheduling training courses and develop an overarching philosophy and strategy for the role of training in capacity building. Are existing courses adequately tailored to the needs of trainees with differing, and changing, roles and situations? What is the optimum timing for trainee follow-up and refresher courses? Do the training resources being used themselves need to be upgraded? Can especially promising trainees be recycled back as occasional trainers themselves? Answering these and other such questions will help the Avert capacity building enterprise better fill gaps in the knowledge and performance of NGOs and service providers. Manuals. Avert has produced a number of manuals to guide implementation of different initiatives and/or their technical components. Technical manuals on STI have been produced for all categories of staff/practitioners. Unfortunately, the STI manuals do not lay emphasis on syndromic case management or treatment. The manual for pharmacists does not mention common drugs, doses or side effects. The manual for non-allopathic practitioners does not contain discussion of the approach to STI in non￾allopathic systems of medicine. A widely praised manual on WPI was prepared by Avert and the Bombay Chamber of Commerce and Industry. An HIV & TB manual is under preparation. Every effort should be made to design manuals to be user-friendly, and to address issues that are peculiar to a particular group. For example, a manual for pharmacists should contain more information on drugs and their side effects and less on counseling. NGO partner development. Avert has a well-developed system for building capacity of NGOs whose applications for support are accepted, through a series of technical and EVALUATION OF THE AVERT SOCIETY PROJECT 25 budgetary reviews, in-house and on-site. Once a partnership is approved and an MOU signed with an NGO, Avert hosts an initiation workshop, and follows up with regular “participatory site learning visits” (PSLV) and “experience-sharing review meetings” (ESRM) over the life of a partnership. These interactions tend to focus largely on administrative matters, and because of high NGO staff turnover there is a constant need to provide training for new staff. Equal attention needs to be paid to regular training and refresher training for all categories of staff on care and support issues. It is suggested that Avert consider starting a bimonthly or quarterly newsletter, edited by the capacity building specialist, to keep partner NGOs abreast of the project’s work and findings. It will help disseminate information on Avert activities in different areas, provide a forum for NGOs to air their ideas and concerns, and generally develop a sense of shared commitment and teamwork. Finally, the team suggests that Avert technical staff should regularly participate in meetings and workshops dealing with administrative matters, and similarly administrative staff should participate in technical workshops. This would help to bring about better synergy between the two wings, a need that is discussed further in this report. RECOMMENDATIONS: • Curricula of training programs contracted by Avert for NGO personnel need to be regularly reviewed and upgraded. Avert technical staff should sit in on trainings on an unscheduled basis to monitor for content and consistency. Trainers should make follow-up visits to trainees. • Counseling training needs a comprehensive review. It is recommended that a counseling officer be added to the Avert C&S office. • Training and technical manuals produced by Avert should be reviewed for content and completeness, and made as user-friendly as possible. • To build teamwork and a shared sense of mission within Avert and among its NGO partners, publication of a regular Avert Society newsletter is recommended, and financial management and technical personnel should participate in each others’ workshops. • Develop an overarching philosophy and strategy for the role of training in capacity building by answering key questions about training quality, flexibility and impact. III.G. RESEARCH / M&E Reality and potential of Avert's research role. The Avert Society was given a state￾level mandate to develop an HIV and AIDS research agenda. Its objective is to develop a high quality database, and use the findings for program planning, advocacy and decision making at the policy level. In 2003, building on earlier work by FHI and others, Avert and the National AIDS Research Institute in Pune jointly organized a one-day meeting to identify research priorities in HIV/AIDS in Maharashtra. These activities helped establish a broad research agenda for the state and the Avert program. Various waves of behavioral surveillance surveys (BSS), and mapping exercises at the state level have also produced useful sets of information. 26 FINAL REPORT In the view of the evaluation team, however, within the current structure and functioning of Avert the research component is not well integrated with other sub￾systems – targeted intervention, care and support, communication and social marketing. All seem to be, by and large, stand-alone components with only informal linkages between them. Research, and the need for useful data, could provide a crucial linkage among all the technical units of Avert. To this end, the Research Unit should be involved in planning and program development exercises, and be urged to undertake more frequent field visits in company with other technical staff. There is a need to build capacity at the NGO level to use qualitative tools and ethnographic approaches to understand the needs of communities that are, or should be, targeted with prevention messages, such as, for example, clients of sex workers. Research suggests that client profiles vary from one CSW site to another. During field visits the team heard that, while at one site students constituted the major clientele, in another it was migrant laborers and pilgrim tourists. Another important piece of information could be a CSW’s client load. Yet another area about which little understanding exists is the STI treatment-seeking behavior of CSWs and their clients. Complex issues also surround hijras, MSMs, bisexual men and, significantly, married women who are in no position to question their husband’s behaviors and negotiate safe sex. Mechanical, “information-driven” interventions will have little impact on the behaviors and lives of those who are at highest risk of acquiring and transmitting the virus. An important distinction should be made between mapping exercises and gathering of baseline information. The current practice of mapping by NGOs yields little more than head counts of target populations. Even these are often unreliable, making them of no use to planning or monitoring. True "social mapping" will yield meaningful ethnographic and qualitative baseline data, and guide program planning. Until recently most NGOs did not clearly distinguish between mapping and baseline information and used them interchangeably. The evaluation team was however informed of plans to identify a set of key behavioral indicators that should be collected, first as part of a baseline exercise and again at the end of the project to track behavior change. The team applauds this effort. Operations research (OR). Missing from the current research agenda of Avert is the recognition of OR as a tool not only to plan, but also to validate what is and is not working in an intervention. As a matter of strategy, implementing partners should be encouraged to systematically think about the key questions they will want to address in their programs, with OR activity developed around those questions. Illustrative such research questions include: • What are factors that contribute to the effectiveness of peer educators and how can they be sharpened? • What are the barriers and facilitators to reaching out to a target population, and how can one be overcome and the other enhanced? What are the alternative strategies for outreach? • How best to integrate alcohol intervention with HIV prevention? What are realistic intervention strategies given existing resources and channels within the community? EVALUATION OF THE AVERT SOCIETY PROJECT 27 • How to reduce loss to STI follow-up and build appropriate and relevant referral systems? What are the most appropriate strategies to achieve a true continuum of care in a community setting? • How to reduce stigma for seeking STI or VCT services and also provide effective counseling services in a community setting? • Which communication strategies are more appropriate and likely to be effective in a given community? Monitoring and Evaluation. Daily diaries, MTRs, participatory site learning visits and experience sharing review meetings all constitute key components of M&E activity. PSLVs and ESRMs in particular are highly appreciated by NGO partners. These are laudable efforts and should be maintained with high quality technical inputs and greater regularity. The practice of routine compilation of MTRs (it has started very recently) at the Avert level should be reviewed carefully to assess its usefulness and ability to provide feedback to programs on the ground. Avert technical staff is making an effort to provide feedback to NGOs on the basis of MTRs, and it would be appropriate to make this practice as effective and responsive as possible as an M&E tool. Avert has made good use of consultants for conducting PSLVs and ESRMs as monitoring mechanisms. However concerns were raised to the evaluation team about the administrative and financial requirements of consultant hiring, which tend to stifle the process in bureaucracy. The team was told that the new Avert administrative systems manual, when put into practice, allows for establishing an approved roster of consultants from which technical staff can draw more efficiently and flexibly than in the past. If this is not the case, the issue should be addressed immediately. Finally, as part of M&E activity, it will help to develop a few key indicators at the NGO level to assess: ƒ Quality of services delivered ƒ Reach and effectiveness of program ƒ Program sustainability A scheme should be developed to collect information on the above areas of program at a regular and reasonable interval. Office configuration and technical assistance. Currently M&E is an independent component within the Avert system. It makes more sense that M&E should be an integral part of the Avert Research Unit. This will ensure much needed synergy between M&E activities and research, and also make it possible for research findings to be used in program planning as well as monitoring. To the extent that technical assistance is sought or required to upgrade the M&E function at Avert, the team recommends looking to Operation Lighthouse/PSI as a possible resource. Operation Lighthouse has developed and is using a highly effective 28 FINAL REPORT approach to baseline development and monitoring of impact that Avert might find applicable to its needs. Were a contractual consulting arrangement to be contemplated for such technical assistance, it seems reasonable to think that it might be funded under the technical component of USAID funding of Avert, as are JHU and HLFPPT. RECOMMENDATIONS: • Under the leadership of the Research Unit, Avert's approach to gathering of baseline information in preparation or support of NGO partner activities needs to be fully reviewed and upgraded. • Operations research should be used more vigorously as a tool to answer important project-related research questions. • Encourage and assist NGOs to use M&E data from their projects as a formal feedback mechanism to revisit and update project goals and strategies. • The monitoring and evaluation function should be integrated with the Avert Research Unit, under the direction of the Research Specialist. Every effort should be made to integrate research and M&E awareness and priorities with other technical functions. • Consider accessing technical assistance for baseline development and monitoring from Operation Lighthouse/PSI. III.H. CONDOM PROMOTION AND SOCIAL MARKETING Status report: Avert has been involved with condom promotion in the past few years, but, as noted earlier, this involved ensuring access for NGO partners and their target groups to free condoms. It is only recently that HLFPPT was engaged to lead a condom social marketing effort under the Avert/USAID technical component. It is a positive step that Avert immediately made resources available for this new involvement, spearheading with HLFPPT the establishment of a Condom Social Marketing Working Group, of which they are members along with MSACS, MDACS and JHU (as consultant). Plans for NGO involvement in this effort are evolving. HLFPPT is presently surveying NGO willingness to participate in CSM, including a training needs assessment. There is some urgency to this effort, since the HLFPPT contract runs only to September 2006, but the evaluation team hopes that it will be extended. NGO partner coordinators and ORWs have been identified as likely recipients of sensitization on the CSM process. Current and projected elements of the CSM program are: ƒ placement of condom vending machines ƒ generic condom promotion campaign ƒ capacity building for innovative retailing ƒ public/private partnership development EVALUATION OF THE AVERT SOCIETY PROJECT 29 ƒ OR on the female condom ƒ introduction of quality condom signage Avert has a dedicated position of condom promotion and social marketing officer, who is the point person for collaboration with HLFPPT, which the evaluation team feels is both effective and forward-looking. Condom supply. As has been noted in this report, supply of free condoms to Avert partners can sometimes be problematic. Steps in the process include: ƒ NGOs estimate annual condom requirement ƒ Request is submitted via Avert to MSACS and the DHS ƒ DHS consolidates requests and sends indent to Health Dept. ƒ District Health Officer supplies condoms to NGOs ƒ NGOs submit periodic utilization reports to Avert/MSACS. Weaknesses in the process include the fact that requests are not usually based on reliable consumption data. Requests are often buried in monthly management information systems (MIS) reports to Avert, and there is no formal feedback system for their tracking. This leads to delays and gaps, which NGOs often fill (when or if they have the means) by acquiring condoms from independent sources. There are also situations whereby, even though requests are timely and based on accurate figures, condoms are not received on account of pipeline problems at State warehouses. These issues can be overcome, at least in part, by building the capacity of NGOs for rational condom forecasting, and instituting a system whereby supply shortfalls are not carried forward to the next month, so as to ensure that new deliveries are on schedule. More generally, a formal feedback system should be instituted whereby the directors of Avert, MDACS and MSACS are kept regularly informed of condom supply issues. An historical analysis of factors contributing to stock-outs and other problems will help the Societies prepare more effectively for future such events, as will the establishment of regional buffer stocks for use in emergencies. In their totality, however, these concerns point up the importance of moving carefully towards a situation in which condom needs are increasingly met through a well-planned and efficiently run social marketing program. RECOMMENDATIONS: • Current limited involvement in CSM needs widening. This must include creation of value association around condoms, and promoting priced condoms. • Long-term, free distribution of condoms is neither sustainable nor desirable. Creating a receptive frame of mind for purchasing condoms also benefits the effect of condom demonstration. 30 FINAL REPORT • It is essential that Avert staff and NGO partners not only understand the social marketing process but also understand its long-term benefits. HLFPPT will play a key role as technical advisor and implementer. • Unique opportunities for strategic introduction of CSM in the work place and at community care centers (and elsewhere) must be fully realized. • Meanwhile, certain steps can be taken, as described, to make the current system of distribution of free condoms more dependable and responsive. . EVALUATION OF THE AVERT SOCIETY PROJECT 31 IV. PROJECT MANAGEMENT Strengths. The Avert Society vision and mission are clear, and the structure for pursuing them is largely in place. Despite an extended period of management turmoil, USAID’s patience in maintaining support has been rewarded, in that the Society's leadership is focused on the future and has hired a committed, well qualified technical staff. The project has been steadily scaling up, in terms of NGO partners accepted for funding, with 54 now operating under MOUs with Avert. That number is expected to approach 100 by March 2006. Contracts with JHU (for communication) and HLFPPT (for CSM), supported under the technical component of USAID's funding of Avert, are in place and ready to be maximized. Avert financial management has recovered from previous difficulties with release of funds from NACO, has cleared overdue audits, and has a new systems manual ready to be launched that will streamline many administrative procedures. Avert has successfully established itself as a stand-alone agency (i.e., not under the parenthood of a Maharashtran NGO), and developed close, collaborative working relationships with MDACS and MSACS, as well as other stakeholders in Mumbai and around the state. There will always be the occasional overlap of strategies and activities. But the fact that Avert, MDACS and MSACS are located in the same Mumbai compound will (among other reasons) ensure effective synergy and coordination of strategies, and thus optimum application of available resources to prevention of the spread of HIV. Organizational culture. Operationally speaking, Avert is highly organized, as illustrated by its Status Document, November 2005, which the evaluation team used as its guide throughout this assignment. Section III.A. of this evaluation report describes the primary and cross-cutting areas of project activity that make up the dynamic Avert Strategic Plan. Targets for which different technical staff are accountable in terms of new NGO partnerships, training exercises, and other activities are clear and specific. The process whereby new partnerships are developed, from application through proposal development to Governing Board approval and subsequent implementation and follow￾up, is comprehensive and well thought out. Its elements (PSLVs, ESRMs, use of MTRs as monitoring tools) are described elsewhere in this report, and it could indeed serve as a model for other agencies of careful nurturing of sub-grantees and their growth. In short, goals and tools for project implementation are all in place. 32 FINAL REPORT On the other hand, there is a palpable need within the Avert staff for team building, and for the leadership necessary to make that happen. Although informal communication within the technical staff is relaxed and cordial, nonetheless the different technical units (targeted interventions, C&S, capacity building, research, etc.) operate in a vertical manner, without much formal programmatic interaction. Technical staff who accompanied the evaluation team to visit NGOs and activities outside of their particular areas of responsibility frequently commented about how much they were learning about areas unfamiliar to them. It seemed to the team that (for example) TI and C&S staff should have greater familiarity with each others’ involvement with NGO partners; the Capacity Building Unit should be more in touch with training needs of different NGO outreach and service programs; and the Research Unit should be closely monitoring everyone’s activities for clues as to their impact. Far more serious is the gulf between the Avert technical program and financial management offices. The former views the latter as rigid and untrusting. The latter views the former as disrespectful of regulations. Not surprisingly, both sides have some reason for their attitudes. In the field of social development, a rigid approach to finance and administrative rules and regulations can stifle enthusiasm, and the technical staff frequently feels this way. At the same time, Avert financial managers have been responsible for overcoming a problem-filled financial management situation, and can be excused for their commitment to rules and regulations so as to avoid future, potentially fatal, disasters. The two can, and must, meet, understanding that they will not always agree but that a willingness to listen to, and reason with, each other is essential to the well-being of the organization. The evaluation team urges a concerted effort at team building, one that will result in more effective, less destructive, collaboration between program and finance. Approaches might include: • Planning a staff retreat, chaired by an outside consultant, to give staff a chance to air their concerns, share their visions for the organization, and begin to establish a greater rapport among all units of the organization. • Making a concerted effort to include people from all units in important programmatic meetings, such as ESRMs and PSLVs. • Getting all staff into the field. With all that they have had to deal with, financial managers have not had the time to get out of the office over the past year or so. Now that things are caught up, participating in site visits will go a long way to giving them a better understanding of field realities. The same emphatically goes for the communication specialist. • Use a particular theme to engage full staff participation. The one that comes most readily to mind is the reorientation of the Avert communication strategy and program. This would afford a chance for everyone to learn not only about a more holistic, BCC approach to communication, but to learn how it will impact each other's particular piece of the action. EVALUATION OF THE AVERT SOCIETY PROJECT 33 Leadership gaps. The responsibility for building a more cohesive, mutually supportive organizational culture, whether through these or other methods, lies first and foremost with the Avert Project Director (PD). The evaluation team came to understand that the incumbent has, consciously or unconsciously, developed a persona of being largely outside of the organization, dealing with issues and stakeholders unconnected to day-to￾day office operations. But he has also made a frequent practice of jumping back into the fray, making pronouncements and issuing orders, sometimes out of context, then leaving again without taking the time to explore issues in depth, or to listen to staff concerns about their implications. The result is uncertainty on the part of the staff as to how much he wants to be involved, how they should relate to him, and whether he values their inputs. There is no question on the part of the staff, nor of the evaluation team, of the PD's commitment to Avert and its work. Hired at a difficult time in the Society's life, he has presided energetically over the process of both “catching up and scaling up,” which has put the project in the promising place it is today. But now it is time to ease off on the throttle, work diligently to build the team cohesion that everyone covets, become a better, more patient listener, and strike a more effective balance between “outside” and “inside” work. It seems to the evaluation team that the PD can use proactive assistance, primarily from USAID, in working out this balance and understanding its importance, and that the possibility of providing him with leadership training should be considered. An alternative to this is for the PD to devote himself exclusively to outside, inter-agency, political and ambassadorial types of activity, and leave project operations entirely under the supervision of the associate project director (APD), with the APD reporting to the PD on a regular basis. This is a structure that is in effect in many organizations, with success. By virtue of his long service with Avert, the APD is widely respected by his staff and in the field, and his understanding of the work and commitment to Avert's mission is unquestioned. In any event, whatever arrangement is decided upon, the evaluation team urges strongly that it be made clear that, in keeping with his title, the APD is the official second-in￾command of the Avert Society, with lead decision-making authority in the absence of the project director. This will remove a nagging ambiguity of responsibility, especially between the APD and the finance manager, that has contributed to internal confusion and to the lack of staff cohesion described above. Staff expansion and development. Several staff additions/changes have been suggested in the text of this report, including: • A communication specialist with health and/or social communication expertise • 2-3 field-based communication coordinators • A counseling officer to be attached to the care and support unit • Combining the M&E function with the work of the Research Unit It also seems logical, considering the steady scaling up of numbers of NGO partnerships, that new program officers should be considered, at least for the targeted interventions and care and support units. 34 FINAL REPORT Finally, attention should be given to helping program staff avail themselves of short￾term training opportunities to build skills and enthusiasm. One example might be to provide the Avert condom promotion officer with training in CSM, possibly to be arranged through/provided by HLFPPT. RECOMMENDATIONS: • USAID should take a firm, proactive role in working with the Avert Project Director on an ongoing basis, to clarify his perception of his role and to strengthen aspects of his leadership style as described above, including leadership training as needed. • USAID should encourage and facilitate essential internal team-building initiatives so as to regain the organizational cohesion desired by all. • Fill new staff positions, as described. EVALUATION OF THE AVERT SOCIETY PROJECT 35 V. PROJECT GOVERNANCE Governing Board. The Avert Society Governing Board (GB), chaired by the Secretary of Health of Maharashtra with a representative of USAID/India serving as Vice Chairperson, provides policy guidance and approves major funding and strategy decisions for the project. MDACS, MSACS and NACO are also represented on the Board. From its discussions and background reading, the evaluation team understood that the GB has not always functioned in the most efficient and supportive manner. Some examples: • In the past, last minute postponement of Governing Board meetings has frequently resulted in serious delays in project implementation, in that GB approval is required for funding of any new NGO partnerships and other major financial decisions. • The GB has frequently required NGOs themselves to make presentations of funding proposals. This is intimidating to them, time-consuming, and, in the team's view, disrespectful of the Avert program staff's ability to present The first obstacle can be obviated by setting, and holding to, a schedule of quarterly Governing Board meetings a year in advance. Staff needs to be able to count on firm dates around which they can plan their proposal development process. With respect to the second issue, a recently adopted practice of having staff present new funding proposals to a Board sub-committee, to save time at full GB meetings, is to be encouraged. In the same vein of streamlining operations, the evaluation team sees no need to increase the size of the Governing Board from its present seven members, as has been discussed and is permitted in the by-laws. The forum of regular quarterly GB meetings presents an excellent opportunity to formally review and monitor efforts on the part of MDACS, MSACS and Avert to avoid duplication of effort among the three Societies, resolve other disputes, and thus maximize their resources in combating HIV statewide. Experience sharing. It is well known that Avert was modeled after the APAC project, and that continuous sharing of experiences was envisaged. This has occurred to some extent, but in an ad hoc manner. APAC has much to teach Avert, and by this time the reverse is probably also true. But any contemplated experience sharing should be built 36 FINAL REPORT around a particular theme, such as NGO capacity building or communication, with goals, a planned schedule of exchange visits, and a timeline. Otherwise it will remain ad hoc, with little lasting value. Elsewhere in this report, the evaluation team has suggested specific instances where experience sharing with Operation Lighthouse/PSI might also be beneficial to Avert. Specific opportunities lie in strengthening interpersonal communication as part of a broad communication strategy, and in developing Avert's research/M&E capacity. USAID oversight. The evaluation team recommends that USAID be firm and proactive in its oversight of the Avert Society project, a role fully justified by its status as GB Vice Chair. Given the amount of money invested in, and the stakes for, the State of Maharashtra, this seems appropriate. The first priority, as discussed above, must be to work with the PD to clarify his role, and to strengthen aspects of his leadership style. The second is to encourage and facilitate internal team-building initiatives so as regain staff harmony and cohesion. Thirdly, to the extent that the recommendations of this evaluation are thought to be actionable, USAID is urged to meet with Avert staff to develop an appropriate action plan and schedule. It should also meet with Avert and JHU to jump-start recruitment and planning for an expanded, more all-encompassing communication program (again, assuming the team's recommendations are accepted). The team leaves to USAID/Delhi a decision as to whether to give the USAID representative in Mumbai a more pronounced oversight role in the absence of someone from the Delhi office. Her ready access to the Avert office is, of course, a convenience. Whether she has the stature, if given a specific mandate (beyond the relatively meaningless role of “coordinating”), to play a useful role is a question that the team cannot answer. RECOMMENDATIONS: • The schedule for Governing Board meetings should be set one year in advance and adhered to, to facilitate staff planning of the grant approval process. • Other steps to streamline GB function are to be encouraged. • The GB must ensure that true synergy is achieved and maintained between Avert, MDACS and MSACS in fulfilling their state-level HIV prevention mandate, in particular by avoiding harmful duplication of effort. • USAID should make the most of its role as GB Vice Chair (and principle funder) in encouraging and overseeing Avert leadership development and team building. EVALUATION OF THE AVERT SOCIETY PROJECT 37 VI. OVERALL RECOMMENDATION The specific technical and programmatic recommendations in this report speak for themselves. Above and beyond them, however, the Avert Society evaluation team highly recommends that USAID make whatever arrangements are necessary (presumably an unfunded extension) to see that the work of the Society continues well beyond its scheduled termination date of September 2006. Given its “burn rate” to date and the size of the original authorization, Avert should be able to continue vigorous operations for three to four more years. This gives it an excellent chance to make a major contribution to slowing the advance of HIV and AIDS in the bellwether State of Maharashtra. EVALUATION OF THE AVERT SOCIETY PROJECT 39 ANNEX A PERSONS AND ORGANIZATIONS CONTACTED National AIDS Control Organization (NACO) (New Delhi) Dr. N.S. Dharmshaktu, Additional Project Director Mr. Shridar, NACO Under-Secretary for Finance USAID/India (New Delhi) George Daikun, USAID Mission Director Beth Hogan, Deputy Mission Director Robert M. Clay, Director, USAID Office of Population, Health & Nutrition (PHN) V. Sampath Kumar, Project Management Specialist, USAID/PHN Sanjay Kapur, Chief, HIV/AIDS and Infectious Diseases Division, USAID/PHN Meri Sinnitt, Deputy Director, USAID/PHN Janet Hayman, USAID/PHN Nutan Zarapkar, Program Management Specialist, USAID/PHN (Mumbai) The Avert Society Vijay Satbir Singh, IAS, Chairman, Avert Governing Board S.M. Sapatnekar, Project Director Vishwanath Koliwad, Associate Project Director Virender Chawla, Finance Manager Vandana Bhatia, Care and Support/STI Specialist Anna Joy, Targetted Intervention Specialist Usha Maheshwari, Capacity Building Specialist Jawahar Joshi, Communication Specialist Jayanta Basu, Research Specialist Tushar Deshmukh, Condom Promotion Officer Reena Sen, Audit Officer Vijay Dhulla, Finance Officer Milan Godse, Administrative Officer N.J. Rathod, Interim Project Director, October 2004 - March 2005 Maharashtra State AIDS Control Society (MSACS) Prakash Sabde, IAS, Project Director Mumbai District AIDS Control Society (MDACS) Nirupa Borges, MD, Project Director 40 FINAL REPORT Hindustan Latex Family Planning Program Trust (HLFPPT) Anu Puri, Maharashtra Condom Social Marketing Project PSI/Operation Lighthouse Sanjay R. Chaganti, Program Director Vivek Sharma, Program Manager, Research Ms. Binitha, Advocacy and Linkages Manager Johns Hopkins University, Center for Communications Programs (JHU/CCP) Sonalini Roy, Country Director Aparna Sah, Project Officer Family Health International (FHI) Kathleen Kay, Country Director (New Delhi) Sanjeev Singh Gaikwad, Associate Director, Maharashtra Avert Society NGO Partners (NGOs are grouped by district and project type. In addition to contacts named, the team met with other NGO staff members, board members, outreach workers, counselors, peer educators, and many members of target groups.) Mumbai District Dai Welfare Society, TI/transgenders Laxmi Narayan Tripathi, President Society for Human and Environmental Development (SHED), TI/migrants Suresh Wadkar, Project Officer Community AIDS and Sponsorship Programme (CASP), TI/slum dwellers Rekha Raje, Project Coordinator Hope Foundation, TI/slum dwellers Deepak Dhobal, Project Coordinator Yuvak Pratishthan, TI/slum dwellers Prasad Indap, Project Coordinator Sunita, Counsellor Ambekar Institute for Labor Studies (AILS), WPI D.B. Gawde, Institute Director and Project Coordinator Nirman College of Social Work, WPI Amnita Paradkar, Project Coordinator Sheela Keluskar, Outreach worker Bharat Petroleum Corporation Ltd., WPI Harshada Patil, Project Coordinator Niramaya Health Foundation, WPI Dr. Janaki Desai, Medical Director Athar Qureshi, Project Manager UDAAN, C&S/DIC Vijay Nair, UDAAN Trustee and Project Coordinator Thane District SAPREM Prison Project, TI/prisoners Prakash Gaikwad, Project Coordinator EVALUATION OF THE AVERT SOCIETY PROJECT 41 Thane Belapur Industries Association, WPI K. Jayadevan, Secretary and Project Coordinator SHAPATH, C&S/DIC Dinesh Dawi, member The Humsafar Trust, TI/MSM Vivek Patil, Project Officer Vivek Anand, CEO FPAI Bhiwandi, TI/migrant workers Dinesh Shimbi, Project Coordinator Dr. Shishagri Rao, President; also VP, Mumbai Branch, FPAI Hindustan Organic Chemicals Ltd. (HOCL) Hospital, Raigad, WPI Sandip Joshi, Project Coordinator K.V. Joseph, industry official Nagpur District Indian Red Cross Society, TI/CSW R.P. Singh, President YMCA, TI/truckers J.P.F.X. Fernandes, General Secretary and Project Coordinator Indian Institute of Youth Welfare (IIYW), TI/migrant workers Manohar Golpewar, President Smita Puranik, Project Coordinator Bhartiya Atim Jadi Sevak Sangh (BAJSS), TI/migrant workers Mrs. Amritaj Joshi, Project Coordinator Comprehensive Rural Tribal Development Program, TI/slum dwellers K. David, Project Coordinator NKPSIMS Lata Mangeshkar Hospital CCC, CCC Sanjay Kapur, Medical Superintendant Milind Bhrushundi, project originator and volunteer Archana Urkude, Project Coordinator Aurangabad District Marathwada Gramin Vikas Sanstha, TI/CSW Mansukh Zambac, President Appa Saheb Ugli, Project Coordinator Sangita, Head Counselor Gram Vikas, TI/CSW Shiv Pure, Project Coordinator SETU Charitable Trust, TI/truckers C.G. Sawant, Project Coordinator Dr. Babasaheb Ambedkar Vaidyakiya Pratishthan, TI/slum dwellers Rajesh Kapse, Project Implementation In-charge Grish Pawar, Project Coordinator Anant Pandhare, Medical Director, Dr. Hedgewar Hospital (project site) Swapnyapurti, NPA+, C&S/DIC Sunita Prakash Kathar, President 42 FINAL REPORT Solapur District Niramaya Arogya Dam, TI/CSW Dr. B. Kinikar, President Mrs. Kinikar, Assistant Director Samata Samajik Vikas Sanstha, TI/CSW Prabhavati Mardikar, Director Solapur Zilla Samajik Karya Samitee, TI/truckers Ulhas Patil, President Jankalyan Samiti, TI/slum dwellers and C&S/CCC J. G. Shelgekar, Director Sangli District MookNayak, TI/MSM Shaligram Kamble, President Santosh Kamble, Coordinator Verala Development Society, TI/slum dwellers Prof. Arun Chauhan, Hon. Secretary Zai Kulkarni, Project Coordinator Yerala Project Society, C&S/CCC/VCTC Prof. S. Saptasagar, President Ms. Subhagi, Coordinator Rajarambapu Dnyan Prabodhini (ASTHA), C&S/HBC Ashok Todkar, Project Officer Aamhich Aamache Sanstha, C&S/DIC Dr. Uday Jagdale, Founder/Advisor Satara District Bel Air Hospital, C&S/CCC Fr. Tomy, Director Pune District John Paul Slum Development Project, C&S/CCC George Swami, President EVALUATION OF THE AVERT SOCIETY PROJECT 43 ANNEX B QUESTIONNAIRE CIRCULATED TO NGO PARTNERS 44 FINAL REPORT QUESTIONNAIRE CIRCULATED TO NGO PARTNERS TO: NGO partners of the Avert Society FROM: The Avert evaluation team At the request of USAID/India and the management of the Avert Society, we have been asked to undertake an evaluation of the work of the Avert project in Maharashtra. Our team of experienced health professionals, whose names are listed below, will, through meetings and discussions with project personnel and visits to project beneficiaries, seek to (1) review the effectiveness of project activities and management systems; (2) assess the project's strengths and weaknesses; and (3) make recommendations as to future directions and priorities. Our goal is to help the Avert Society maximize its technical and material resources in implementing HIV prevention and care activities in Maharashtra. The field portion of the evaluation will take place over one month, beginning the week of November 14. The evaluation team looks forward very much to meeting and talking with as many Avert NGO partners and other stakeholders as possible. We have much to learn from you as we assess the project's role on the state and national scene. To give us a grounding in the work of Avert, an understanding of the issues you as its partners face in project implementation, and your thoughts on future directions, we are asking Avert partners to complete the brief questionnaire that accompanies this message. Information provided will help us prioritize issues and plan site visits through which to explore project activities in depth. We would appreciate very much your taking the time to respond to these questions, which we have left largely open-ended so you can answer in a manner that best suits your agency's situation. Completed questionnaires should be returned before November 7th. They will be used solely by the evaluation team for its own information and planning purposes, and all responses will be held in strictest confidence. We look forward to meeting and talking with as many of you as possible in the course of our evaluation. With thanks and best wishes, The Avert Project Evaluation Team D.K. Bose, Communications/Social Marketing Specialist Eliot T. Putnam, Jr., Team Leader/Management Specialist Gherad D. Ravindran, Care and Support/VCT Specialist Ravi K. Verma, Prevention/NGO Specialist EVALUATION OF THE AVERT SOCIETY PROJECT 45 Avert Society Evaluation Questionnaire for NGO partners (and other stakeholders) (Please submit completed questionnaires by e-mail, as indicated. Responses are for the exclusive use of the Avert evaluation team. Information provided will be held in strict confidence, without attribution of specific comments to individual respondents.) 1.) Briefly describe the mission and work of your agency in HIV prevention and care, and the groups to which your activities are targeted. 2.) Describe the support provided to your NGO by the Avert Society. Is it financial, technical, managerial, logistical or some combination of these?\ When did your partnership with Avert begin, and how long is it expected to last? 3.) Describe how the support you have received from Avert has helped develop the capacity of your agency to fulfill its mission. What can you do now that you were not equipped to do before your partnership with Avert was established? 46 FINAL REPORT 4.) Has the support that your NGO has received from Avert met all of your expectations? If not, please explain how that support might have been more effective, and/or what have been the obstacles to complete fulfillment of the partnership. If yes, what were the factors that made utilization of Avert support most effective? 5.) Based on your experience as an Avert partner, please list what you consider to be three of Avert's greatest strengths, and three areas where you feel improvement is needed. Strengths: 1. 2. 3. Areas needing improvement 1. 2. 3. 6.) Looking to the future, how do you hope to strengthen and/or expand the mission of your NGO, and how do you expect Avert to contribute to that process? Name of Avert partner: EVALUATION OF THE AVERT SOCIETY PROJECT 47 ANNEX C AVERT SOCIETY EVALUATION SCHEDULE Evaluation of Avert Society by a team consisting of: 1) Mr. Eliot T. Putnam (Mobile No. 9833532516) 2) Dr. Ravi Verma (Mobile No. 09810595578) 3) Dr. Ravindran (Mobile No. 9880081770) 4) Mr. D. K. Bose (Mobile No. 9820072539) PROGRAM SCHEDULE Date & Time Details of Visit Person Responsible from Avert 17TH NOVEMBER 2005 11.00 hrs to 13.00 hrs Introduction & Presentation on Avert Society and discussion on issues arising out of presentations Dr. S. M. Sapatnekar 13.00 hrs to 14.00 hrs Lunch 14.00 hrs to 17.00 hrs Presentation and discussion continued 18TH NOVEMBER 2005 09.30 hrs to 16.00 hrs Discussion with individual technical staff of Avert Society 21ST NOVEMBER 2005 10.00 hrs to 11.00 hrs Preparing schedule for visit to NGOS Mr. Vishwanath Koliwad 11.00 hrs to 12.00 hrs Discussion with PD/APD, MDACS Dr.S.M. Sapatnekar/ Mr. V. Koliwad 12.30 hrs to 13.30 hrs Lunch 13.30 hrs to 15.00 hrs Discussion with Country Director & other staff of JHU Dr.S.M. Sapatnekar/ Mr. V. Koliwad 15.30 hrs to 16.30 hrs Discussion with Project Director, FHI Dr.S.M. Sapatnekar/ Mr. V. Koliwad 48 FINAL REPORT Date & Time Details of Visit Person Responsible from Avert 22ND NOVEMBER 2005 10.00 hrs to 11.00 hrs Discussion with HLFPPT Dr.S.M. Sapatnekar/ Mr. V. Koliwad 11.30 hrs to 12.30 hrs Discussion with PD, MSACS Dr.S.M. Sapatnekar/ Mr. V. Koliwad 12.30 hrs to 13.00 hrs Discussion with Dr. N. J. Rathod, DHS 13.00 hrs to 14.00 hrs Lunch 14.00 hrs to 14.30 hrs Discussion with Avert Staff on Planning for visits 14.30 hrs to 15.30 hrs Discussion with Project Director, PSI Dr.S.M. Sapatnekar 23RD NOVEMBER 2005 08.00 hrs to 11.30 hrs Visit to ‘Nirman” – WPI Project Discussion at Goregoan, Extension Training Centre - Visit to Naka Sites 11.30 hrs to 12.00 hrs Journey to UDAAN Secretariat Naka sites to Ghatkopar 12.00 hrs to 13.30 hrs UDAAN – Observe activities of DIC and discussion with Staff 13.30 hrs to 14.30 hrs Lunch 14.30 hrs to 15.30 hrs Ghatkopar to Vileparle 15.30 hrs to 17.30 hrs Hope Foundation – Observe activities and discussion with Staff Ms. Anna Joy (Mobile No.: 9892269680) 24TH NOVEMBER 2005 Team: Mr. Eliot T Putnam & Dr. Ravi Verma 08.30 hrs Start from Hotel to FPI, Bhiwandi 10.30 hrs to 13.00 hrs FPAI, Bhiwandi - Observe migrant workers Project (TI) and discussion with Staff 13.00 hrs to 13.30 hrs Bhiwandi to Kalyan 13.30 hrs to 14.00 hrs Lunch 14.30 hrs to 15.30 hrs Kalyan - Visit to “Shapath” - Observe activities and discussion with Staff 15.30 hrs to 16.00 hrs Travel to “SAPREM” 16.00 hrs to 17.00 hrs SAPREM - Observe activities and discussion with Staff 17.00 hrs Kalyan to hotel Dr. Vandana Bhatia (Mobile No.: 9820263746) Team: Dr. Ravindran & Mr. D. K. Bose 08.30 hrs Hotel to “SHED” at Dharavi 09.00 hrs to 13.00 hrs Visit to SHED – Observe activities and discussion with staff 13.00 hrs to 14.00 hrs Lunch 14.00 hrs to 14.45 hrs Dharavi to Govandi 14.45 hrs to 15.45 hrs Niramaya Health Foundation - Observe activities and discussion with staff 15.45 hrs to 16.00 hrs Niramaya Health Foundation to Dai Welfare Trust 16.00 hrs to 16.45 hrs Dai Welfare – Observe activities and discussion 16.45 hrs to 17.45 hrs Govandi to Kalyan (Humsafar) Ms. Usha Maheshwari EVALUATION OF THE AVERT SOCIETY PROJECT 49 Date & Time Details of Visit Person Responsible from Avert 17.45 hrs to 18.30 hrs Humsafar Trust - Observe activities and discussion with staff 18.30 hrs On the way back to hotel observe the MSM sites. 25TH NOVEMBER 2005 Team: Dr. Ravindran & Dr. Ravi Verma 08.30 hrs Start from Hotel to CASP (Goregaon) 09.30 hrs to 12.30 hrs CASP – Observe activities and discussion with staff 12.30 hrs to 13.30 hrs Lunch 13.30 hrs to 14.30 hrs Goregaon to Chembur 14.30 hrs to 16.00 hrs Visit to BPCL Refinery – Observe activities and discussion with staff 16.00 hrs to 16.30 hrs Travel from Chembur to Mulund 16.30 hrs to 17.30 hrs Yuvak Pratishthan - Observe activities and discussion with staff Mr. Tushar Deshmukh (Mobile No.: 9820294563) Team: Mr. Eliot Putnam & Mr. D. K. Bose 08.30 hrs Start from Hotel to HOCL (Panvel) 10.00 hrs to 12.30 hrs HOCL - Observe activities and discussion with staff 12.30 hrs to 13.00 hrs Lunch 13.00 hrs to 14.00 hrs Travel from HOCL to TBIA 14.00 hrs to 15.30 hrs TBIA - Observe activities and discussion with staff 15.30 hrs Start from TBIA for AILS (Parel) 16.30 hrs to 17.30 hrs AILS - Observe activities and discussion with staff Ms. Usha Maheshwari 26TH NOVEMBER 2005 & 27TH NOVEMBER 2005 - HOLIDAY 28TH NOVEMBER 2005 TO 29TH NOVEMBER 2005 Team : Mr. Eliot T. Putnam and Dr. Ravi Verma Visit to Nagpur NGOS at Nagpur 1) Indian Red Cross Society 2) YMCA 3) Indian Institute of Youth Welfare 4) Bhartiya Adim Jati Sevak Sangh 5) Comprehensive Rural Tribal Devpt 6) NKPSIMS Lata Mangeshkar Hospital Mr. Tushar Deshmukh Team : Dr. Ravindran and Mr. D. K. Bose Visit to Solapur 1) Niramaya Arogya Dham 2) Samata Samaji Vikas Sanstha, Barsi 3) Solapur Zilla Samajik Karya Samitee 4) Jankalyan Samiti Dr. Vandana Bhatia and Mr. Vishwanath Koliwad (Mobile No.: 9820940288) 30TH NOVEMBER 205 TO 2ND DECEMBER 2005 Team : Mr. Eliot T. Putnam and Dr. Ravi Verma Visit to Aurangabad 1) Marathwada Gramin Vikas Sanstha 2) Gram Vikas 3) Setu Charitable Trust Ms. Anna Joy 50 FINAL REPORT Date & Time Details of Visit Person Responsible from Avert 4) Dr. Babasaheb Ambedkar Vaidyakiya S. 5) Swapnyapurti, NAP Plus 30TH NOVEMBER 2005 TO 2ND DECEMBER 2005 Team : Dr. Ravindran and Mr. D. K. Bose Visit to Sangli 1) Mooknayak 2) Verala Devpt. Society 3) Yerala Project Society 4) Rajarambapu Dynan Prabodhini (ASTHA) 5) Aamhich Aamache Sanstha Visit to Satara 1) Bel-Air Hospital, Panchgani Mr. Vishwanath Koliwad EVALUATION OF THE AVERT SOCIETY PROJECT 51 AVERT SOCIETY, MUMBAI CONTACT NUMBERS OF EVALUATION TEAM MEMBERS: Sr. No. Name Contact No. 1. Mr. Eliot T. Putnam 9833532516 2. Dr. Ravi Verma 09810595578 3. Dr. Ravindran 9880081770 4. Mr. D. K. Bose 9820072539 AVERT STAFF INVOLVED 1. Mr. Vishwanath Koliwad 9820940288 2. Ms. Anna Joy 9892269680 3. Dr. Vandana Bhatia 9820263746 4. Mr. Tushar Deshmukh 9820294563 EVALUATION OF THE AVERT SOCIETY PROJECT 53 ANNEX D Scope of Work: This scope of work (SOW) is for an evaluation of the Avert Project, to assess the programmatic gaps, the effectiveness of the systems and processes and recommendations for future directions and priorities of the Project. This will be an external review through the TASC2-GH IQC. The important stakeholders of the Avert Project will be involved in the planning and the implementation of the evaluation as appropriate. The evaluation team and team expertise requirements are suggested in Annexure 1. The objectives of this evaluation are: 1. Analyze the program coverage and gaps and the effectiveness of the project activities including the quality of the services. 2. Assess the effectiveness of the project management systems including planning, monitoring, finance and procurement. 3. Assess the programmatic coordination of various partners for HIV prevention in the state of Maharashtra. 4. Provide recommendations on the future directions of the Project and how the implementation could be improved and scaled up. The key audience of the evaluation includes USAID/India, USAID/Washington, Avert Society, National AIDS Control Organization (NACO) and Government of Maharashtra. Statement of Work: Tasks: Key areas of the Evaluation The specific tasks of the evaluation team will be as follows: 1. Program analysis: The evaluation team will assess the effectiveness of the project in terms of the coverage of target population with provision of prevention and care and support services including quality of interventions. The team will analyze the appropriateness of the strategies and whether they were complementary and mutually reinforcing. They need to assess the role of Avert Society in identifying and strengthening the capacity of various institutions 54 FINAL REPORT including establishing linkages with relevant stakeholders in public and private sector for providing a sustainable response for HIV prevention. The consultants will identify the level of engagement and support provided by NACO and Government of Maharashtra to the Project. The specific questions which should be addressed but are not limited to include the following: • Assess the appropriateness of the project strategies to achieve the objectives of the project. Does the strategy of the project (prevention, care and support and treatment) provide a comprehensive response and what are the bottlenecks? • Is the coverage of targeted intervention activities adequate in the seven districts? Are their high risk groups that are underserved and requires scaling up? • What are the impediments to scale up care and support activities including VCT services and recommendation for scaling up. • Document the experiences /assess the effectiveness of the workplace intervention models and suggest ways to scale up to the entire state. • Review the progress made in scaling up the state level communication and condom social marketing program marketing activities including coordination with Government of Maharashtra and other partners. • Assess the effectiveness of the capacity building activities of the Avert Project including capacity building of NGOs, health care providers on STI services and state and municipal agencies on HIV programming. Suggest strategies for effective implementation and scale up. • What are the systems developed at various levels including establishing linkages with key stakeholders and strengthening the capacity of various institution for having a sustainable response for HIV prevention? • Assess the cost effectiveness and sustainability of the various programmatic activities and make recommendations for strengthening and scaling up the activities. 2. Project Management: The team will evaluate the effectiveness of the project management systems in the delivery of quality HIV prevention services. The team will assess the adequacy of the management systems in improving the efficiency of the implementation of activities. The team will comment on the management structure of the project and how effective was the structure. The team will also assess the systems for coordination of all technical component activities with the Avert project. Questions that should be addressed but are not limited to include the following: • How effective is the Avert Society model and the management structure to provide a sustainable response for reducing the transmission of HIV/AIDS in the state of Maharashtra? EVALUATION OF THE AVERT SOCIETY PROJECT 55 • Assess the role played by the Avert Society’s Governing Board (GB) in guiding the policies and the progress of the project. How effectively has the Avert GB leveraged/utilized the resources of the Maharashtra Government’s health infrastructure? • Are the project management systems adequate including monitoring and evaluation to strengthen and scale up of the Project activities? What are the areas that require strengthening? • What is the analysis of the staff structure and recommendations for consolidating the staff strengths? • What systems are established in the Avert Project to integrate the technical component activities into the overall response of the Avert Project to HIV prevention? • Analyze the effectiveness of the planning and decision making processes in the Project. How they can be improved? • Assess the financial management systems of the Avert Project including disbursement mechanisms? Review the fund flow mechanism from NACO to the Avert Project. 3. State Level Coordination and Synergies in HIV prevention: The team will assess the level of coordination between the various partners to maximize resources by complementing with each other to avoid duplication of efforts. The team will comment on the systems in place to effect the coordination. The specific questions to be answered but are not limited to include the following: • Are there geographic and programmatic duplication in the implementation of HIV prevention programs in the state? • Are the systems adequate to ensure coordination and synergies of all the HIV prevention efforts? Make recommendations to strengthen the state level coordination. • How effective is the coordination between Avert and other USAID partners? • How effectively is Avert coordination with state AIDS societies and the Government of Maharashtra? Make recommendations to improve the coordination. U. S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov