AVERT END OF PROJECT EVALUATION REPORT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AVERT END OF PROJECT EVALUATION This document was submitted by Social Impact, Inc. (SI), with subcontractor Management Systems International (MSI), to the United States Agency for International Development. It was prepared by Ruth Hope (SI), Deepak Khismatrao (SI), Keerti B. Pradhan (SI) and Barbara Spaid (MSI), under USAID Evaluation Services IQC – Contract No. RAN-I-00-09-00019, Task 0rder No. AID-386-TO-l 0-00003. AVERT END OF PROJECT EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AVERT End of Project Evaluation i This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc to the United States Agency for International Development under USAID Evaluation Services IQC – Contract No. RAN-I-00-09-00019, Task 0rder No. AID-386-TO-l 0-00003. AVERT End of Project Evaluation i ACKNOWLEDGMENTS The evaluation team thanks all the many persons from senior government officers to NGO staff and community members who assisted the evaluation in Delhi, Mumbai and the five Avert districts of Maharashtra. We appreciate the time provided by all the key respondents for answering our questions—we realize everyone of the respondents had taken time out for us from busy schedules—and the thoughtfulness of the responses to our enquiries. We found the briefings by Avert Society staff helpful and value the additional time and effort Avert staff spent responding to our many inquires. Avert advice on our evaluation schedule greatly increased the number of programs we were able to visit and maximized the limited time we had in the field. Avert’s support with booking of accommodation and transportation smoothed our travel throughout the five Avert districts of Maharashtra. Avert District Managers spent long days with the team guiding us to organizations and programs in their districts, making introductions and ensuring necessary protocols were observed. Finally, the international consultants appreciate the international travel, logistics, and professional guidance provided by Social Impact. The Consultant Evaluation Team May 2011 AVERT End of Project Evaluation iii ABBREVIATIONS AND ACRONYMS AIDS Autoimmune Deficiency Syndrome ANC Antenatal Clinic ANM Auxiliary Nurse Midwife APD Associate Project Director ART Antiretroviral Therapy ASHA Accredited Social Health Activist AWW Aaganwadi Worker BB Brothel-Based BCC Behavior Change Communication BMGF Bill & Melinda Gates Foundation BSS Behavioral Surveillance Survey CBO Community-Based Organization CCC Community Care Center CCCU Correct and Consistent Condom Use CCDT Committed Communities Development Trust CSW Commercial Sex Worker DAPCU District AIDS Prevention and Control Unit DIC Drop In Center DPM District Project Manager DPO District Project Officer FGD Focal Group Discussion FSW Female Sex Worker GB Governing Board GFATM Global Fund to fight AIDS, Tuberculosis and Malaria GOI Government of India GOM Government of Maharashtra HIV Human Immunodeficiency Virus HSS HIV Sentinel Surveillance ICTC Integrated Counseling and Testing Center IEC Information Education and Communication IDU Injecting Drug User IGA Income Generating Activity IQC Indefinite Quantity Contract KHPT Karnataka Health Promotion Trust KII Key Informant Interview LW(S) Link Worker (Scheme) MARP Most At-Risk Population MDACS Mumbai District AIDS Control Society M&E Monitoring and Evaluation MIS Management Information System MSACS Maharashtra State AIDS Control Society MOU Memorandum of Understanding MSM Men who have Sex with Men NACO National AIDS Control Organization NACP National AIDS Control Program NBB Non-Brothel-Based NGO Non-Governmental Organization NRHM National Rural Health Mission ORW Outreach Worker OVC Orphans and other Vulnerable Children AVERT End of Project Evaluation iv PD Project Director PE Peer Educator PEFAR US President’s Emergency Plan for AIDS Relief PLHIV People Living with HIV PMTCT Prevention of Mother-To-Child Transmission [of HIV] PMP Performance Monitoring Plan PO Project Officer PRI Panchayat Raj Institutes PSS Prerana Samajik Sanstha [a CBO] QA/QI Quality Assurance / quality improvement Rs Rupees [currently $1.00 = Rs 43.50] SACS State AIDS Control Society S&D Stigma and Discrimination SHG Self-Help Group STI Sexually Transmitted Disease STRC State Training and Resource Center TI Targeted Intervention TPM Team Planning Meeting TSU Technical Support Unit UNICEF United Nations Children’s Fund USAID U.S. Agency for International Development VPL Voluntary Peer Leader WPI Work Place Initiative AVERT End of Project Evaluation v CONTENTS ACKNOWLEDGMENTS ........................................................................................................... i ABBREVIATIONS AND ACRONYMS ................................................................................iii EXECUTIVE SUMMARY.......................................................................................................viii I. INTRODUCTION ................................................................................................................... 1 II. BACKGROUND AND PURPOSE..................................................................................... 2 AVERT PHASE I..................................................................................................................... 2 AVERT PHASE II................................................................................................................... 2 CHANGES TO THE PROJECT.......................................................................................... 3 THE EVALUATION PURPOSE AND OBJECTIVES .................................................. 3 III. EVALUATION APPROACH AND METHODS .......................................................... 5 THE EVALUATION TEAM ................................................................................................ 5 EVALUATION APPROACH............................................................................................... 5 EVALUATION METHODOLOGY................................................................................... 5 EVALUATION SCHEDULE ............................................................................................... 6 IV. FINDINGS.............................................................................................................................. 7 1) ASSESSMENT OF THE EFFECTIVENESS OF THE AVERT PROJECT IN TERMS OF PROCESS, OUTCOMES AND IMPACT.................................................... 7 2) ASSESS THE SUCCESS OF THE PROGRAM INTERVENTIONS AND MODELS FOR PREVENTION AND CARE DEVELOPED BY THE AVERT PROJECT.................................................................................................................................15 3) CAPACITY & SCALE-UP OF TARGETED INTERVENTION PROGRAMS .22 4. ASSESS THE EFFECTIVENESS OF THE AVERT SOCIETY VARIOUS MANAGEMENT SYSTEMS—SUCH AS GRANTS MANAGEMENT, FINANCE AND MONITORING AND EVALUATION—IN IMPLEMENTING PREVENTION, CARE AND TREATMENT PROGRAMS .......................................24 V. DISCUSSION OF THE THREE OVERARCHING EVALUATION QUESTIONS & CONCLUSIONS ....................................................................................................................34 VI. LESSONS LEARNED........................................................................................................38 GOVERNANCE AND FINANCING..............................................................................38 PROJECT MANAGEMENT AND OVERSIGHT ........................................................38 PROJECT IMPLEMENTATION ......................................................................................38 VII. RECOMMENDATIONS .................................................................................................40 AVERT End of Project Evaluation vi APPENDICES ANNEX A: DELIVERY ORDER STATEMENT OF WORK...................................III ANNEX B: PERSONS CONTACTED FOR THE EVALUATION.......................XV ANNEX C: TOOLS DEELOPED FOR THE EVALUATION..............................XXI 1. KEY INFORMANT INTERVIEW QUESTIONS FOR NACO ........................XXI 2. KEY INFORMANT INTERVIEW QUESTIONS USAID AVERT MANAGER ..............................................................................................................................................XXII 3. KEY INFORMANT INTERVIEW QUESTIONS AVERT BOARD CHAIRPERSON .............................................................................................................XXIII 4. KEY INFORMANT INTERVIEW QUESTIONS MAHARASHTRA STATE AIDS CONTROL SOCIETY......................................................................................... XXV 5. KEY INFORMANT INTERVIEW QUESTIONS MUMBAI DISTRICT AIDS CONTROL SOCIETY...................................................................................................XXVI 6. KEY INFORMANT INTERVIEW QUESTIONS FHI [BMGF]...................XXVII 6. KEY INFORMANT INTERVIEW QUESTIONS FHI [BMGF]...................XXVII 7. KEY INFORMANT INTERVIEW QUESTIONS PATHFINDER.............XXVIII 8. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY PROJECT DIRECTOR & ASSOCIATE PROJECT DIRECTOR............................................XXIX 9. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON M&E ..............................................................................................................................................XXX 10. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON PROGRAM.......................................................................................................................XXXI 11. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY TSU STAFF............................................................................................................................ XXXIII 12. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON NGO CAPACITY BUILDING.................................................................................XXXIV 13. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON NGO SUB-GRANTING .............................................................................................XXXV 14. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY FOR GOVERNMENT OFFICERS IN THE FIELD....................................................XXXVI 15. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY NGO STAFF...........................................................................................................................XXXVII 16. BENEFICIARY GROUP DISCUSSION AREAS:..................................... XXXVIII ANNEX D: EVALUATION CALENDAR ............................................................... XXXIX ANNEX E: CORRECT AND CONSISTENT CONDOM USE ................................. XLI ANNEX F: HIV PREVALENCE IN AVERT PRIORITY DISTRICTS.................XLIII ANNEX G: IMPROVED PE/VCL PERFORMANCE ................................................XLV ANNEX H: COMPARISON OF PERFORMANCE IN AVERT DISTRICTS TO NON-AVERT DISTRICTS .................................................................................................. LIII ANNEX I: AVERT SOCIETY’S PROJECT DIRECTORS........................................... LIX AVERT End of Project Evaluation vii FIGURES AND TABLES FIGURE 1: MAP OF MAHARASHTRA............................................................................ 4 TABLE 1: SUMMARY OF AVERT INTERVENTIONS ............................................... 6 GRAPH 1: INCREASING CONSISTENT CONDOM USAGE AMONG MARPs . 7 GRAPH 2: HIV PREVALENCE AT STI SITES.............................................................10 TABLE 2: COMMUNITY MOBILIZATION CHARACTERISTICS OF AVERT CBOs.........................................................................................................................................10 FIGURE 2: COMPARISON OF AVERT vs. NON-AVERT DIVISIONS TRENDS IN ICTC TESTING ..............................................................................................................13 FIGURE 3: COMPARISON OF AVERT vs. NON-AVERT DIVISIONS TRENDS IN HIV TESTING.................................................................................................................13 FIGURE 4: CONCEPTUAL MODEL FOR AVERT’S DISTRICT COMPREHENSIVE PROGRAM......................................................................................15 FIGURE 5: CONCEPTUAL FRAMEWORK FOR AVERT’S INTEGRATED CARE MODEL.......................................................................................................................17 TABLE 3: AVERT MIGRANT PROGRAM COVERAGE IN ITS FIVE PROGRAM DISTRICTS ......................................................................................................19 TABLE 4: PROGRESS ON KEY INDICATORS AGAINST INCREASE IN THE COVERAGE...........................................................................................................................15 FIGURE 6: CONCEPTUAL MODEL FOR AVERT’S DISTRICT COMPREHENSIVE PROGRAM......................................................................................15 ANNEX D: EVALUATION CALENDAR........................................................... XXXIX ANNEX E: CORRECT AND CONSISTENT CONDOM USE............................. XLI BROTHEL BASED FSWS (BB-FSWS)................................................................... XLI NON BROTHEL BASED FSWS (NBB-FSWS) ................................................... XLI MEN WHO HAVE SEX WITH MEN..................................................................XLII ANNEX F: HIV PREVALENCE IN AVERT PRIORITY DISTRICTS......XLIII ANC PREVALENCE..............................................................................................XLIII FSW PREVALENCE...............................................................................................XLIII HIV PREVALENCE AT STI SITES....................................................................XLIII ANNEX F: HIV PREVALENCE IN AVERT PRIORITY DISTRICTS ............XLIII ANC PREVALENCE..............................................................................................XLIII FSW PREVALENCE...............................................................................................XLIII HIV PREVALENCE AT STI SITES....................................................................XLIII ANNEX G: IMPROVED PE/VCL PERFORMANCE FOLLOWING ANNEX H: COMPARISON OF PERFORMANCE IN AVERT DISTRICTS.... LIII ANNEX I: AVERT SOCIETY’S PROJECT DIRECTORS...................................... LIX AVERT End of Project Evaluation viii EXECUTIVE SUMMARY The Avert Project was launched in 2001 to support the implementation of the National AIDS Control Program. It works in collaboration with the Maharashtra State AIDS Control Society. Maharashtra is one of the six Indian States in which the prevalence of HIV in adults was higher than 1%. In 2002, the prevalence rate was 1.2%, but in 2008 the HIV Sentinel Surveillance data found the rate had fallen to around 0.25% in the rural population and 0.75% in urban areas. The second phase of the Avert Project was approved for the period from October 2006 to June 2011. The goals for this period were to: i.) scale-up prevention activities that would support saturated coverage (85-90%) of the most at-risk populations (MARPs) in five high-prevalence districts ii.) demonstrate models in community mobilization activities and increase the uptake rate of various care and treatment services in those districts iii.) scale-up work-place intervention in the entire state of Maharashtra. USAID/India requested this end-of-project evaluation to: 1) Assess the effectiveness of the Avert Project in terms of process, outcomes and impact: a) Saturating coverage of MARPs b) Improving quality of services c) Increasing consistent condom usage d) Reducing HIV prevalence 2) Assess the contributions of the Avert Project in developing HIV prevention programs for migrant populations; 3) Assess the effectiveness of the various management systems such as grants management, finance, the monitoring and evaluation of implementing prevention programs, care, and treatment programs; 4) Document the programmatic challenges and lessons learned in implementing the Avert Project; 5) Make suitable evidence-based recommendations for the future directions of USAID HIV/AIDS support in the state of Maharashtra. The evaluation was undertaken between March and May 2011 by a team of two international evaluators and two local evaluators. The evaluation was conducted through an in-depth study of Avert through review and analysis of project and relevant Maharashtra health sector documentation, program visits, key informant interviews and beneficiary group discussions, with the intention of achieving broad and representative involvement of as many Avert stakeholders as possible. Three methods of data collection, triangulation and verification were employed during the evaluation. The team split into two gender-balanced sub-teams, each with an international and a local evaluator for the field evaluation activities. The field evaluation was a subjective, qualitative evaluation of program implementation with approach and tools designed to achieve maximum triangulation of findings within the limitations of the assignment. This report does not represent a quantitative evaluation. AVERT End of Project Evaluation ix FINDINGS 1) Assessment of the effectiveness of the Avert project in terms of process, outcomes and impact in: A.Saturating coverage of MARPs through scaling-up of targeted interventions To date, Avert has almost achieved its target for 85-90% saturated coverage. The current figure stands at 76%. B.Increasing Consistent Condom Usage Among MARPs Behavioral indicators tracked by four separate Behavior Surveillance Surveys (BSS) since 2004 reflect a fairly consistent improvement in Consistent Correct Condom Use (CCCU) among both brothel-based (BB) and non-brothel-based (NBB) female sex workers (FSWs). There was also improvement among men who have sex with men (MSM) in those Avert priority districts that were included in the BSS sample. There is, however, considerable difference in CCCU between paying and non-paying partners among both FSWs and MSM. C.Improving the Quality of Services Avert has no M&E indicators to track quality of services, and there was no written directive from USAID to the Avert Society to ensure that quality assurance (QA) be incorporated into the program. Therefore, there was not a visible focus or concerted effort on QA and quality improvement (QI). As a result, there has been no monitoring of the effectiveness of the behavior change communication (BCC), the backbone of the Avert sub-grant interventions. D.Reducing HIV Prevalence At this point, the data on HIV prevalence are too insufficient and inconclusive to clearly determine whether or not Avert has had an impact on lowering HIV prevalence in its priority districts. If the 2010 and 2012 HIV Sentinel Surveillance surveys should demonstrate a strong downturn in HIV among FSWs, MSM and STI sites in the priority districts, then there would be compelling evidence to support the argument that Avert has indeed had a major influence on HIV prevalence. However, the 2010-2012 data was not released in time to be used for this evaluation. E . How successful were the efforts of the Avert Project in developing community-based organizations (CBOs) and empowering MARPs to access services? Members from all four CBOs visited by the evaluation team said significant benefits had accrued to their community members from the advocacy work they carried out through “pressure groups”. The CBO members mentioned the importance of having a platform for sharing information and supporting each other, improved crisis management, increased access to social entitlements and increased self-respect and confidence. Many CBO members have started to access social entitlements they were previously unaware of, including ration cards, payments for widows and abandoned women, school fee scholarships for their children, and life insurance. F. How Effective Was the Avert Project in Developing a Strong Peer Education Program to Deliver Prevention and Care Services at The Community Level? As of March 2011, there were 388 active peer educators (PEs) working with FSW, MSM and Injecting Drug User communities under Avert. Of these, 303 were trained through Avert. Of the 1710 voluntary peer leaders (VPLs) working with migrant populations, 302 were trained. The PEs work within their communities, providing information on HIV AVERT End of Project Evaluation x and STI transmission, condom use and safe sex practices. A major concern is the lack of a consistent and articulated QA/QI system for monitoring and improving on an on￾going basis the quality of PE/VPL activities, including BCC. G. Has the Avert Project Made a Difference in Priority Districts, Especially Aurangabad and Nagpur? Performance in Avert districts is positive and similar to what is taking place in non-Avert districts where other donors and MSACS provide support. Avert has made a difference in Aurangabad, Nagpur, Jalna, Solapur and Thane in terms of expanded coverage of MARPs and increased uptake of services, particularly in HIV testing by 2010-11, due largely to the strong network of PEs and the successful mobile ICTC vans introduced in 2010. 2) Assessment of the success of the program interventions and models for prevention and care developed by the Avert Project: A. The district comprehensive prevention, care support and treatment program, in scaling up access to quality prevention and care services by MARPS and their partners. What have been the challenges? Key aspects of the comprehensive model include referrals for HIV testing, STI diagnosis and syndrome management, and regular follow up with high-risk persons and those who test positive for HIV to ensure access to a continuum of care, support and treatment. The main challenge to Avert in implementing its model has been repeated changes in NACO policies, priorities, and direction. Evaluation of the district comprehensive model is a challenge as there is no operations research framework in place and no baseline data has been collected. B. The Integrated Care Model Providing Care and Treatment Services to Adult and Children Infected and Affected by HIV/AIDS. What has been the Challenge for Integrating Adult and Child care? The integrated care model aims to provide integrated care services for adults and children infected and affected by HIV/AIDS. Services revolve around the drop in center (DIC) and support groups, with links to the ART center and community care center, and to social entitlements such as ration cards, ideally with peer support and counseling. Challenges include: a paucity of provisions for children, including orphans, affected by HIV/AIDS; meeting the needs of discordant couples and monitoring couples over time to record if they remain discordant. Many of the networks of PLHIV are collectives with elected officers serving fixed terms, presenting a challenge for capacity building, as training is given to hired project staff running the DIC who are simply operating under the name of the network of PLHIV. DIC services are only available to PLHIV living in the district center. C. HIV Prevention for Migrant Populations. What Have Been the Challenges? Of the 1710 VPLs currently working with migrant programs, only 302 have been trained. The evaluation found a clear gap between the VPLs’ level of knowledge and some of the beneficiaries who had misconceptions about HIV transmission and prevention. Coverage of migrants doubled between 2009-10 and 2010-11, but there is still considerable room for further achievement in service uptake for STI visits and HIV testing. Other challenges include barriers to migrants accessing services because of long work hours and fatigue from hard physical labor. Because migrants are not a stable population, it is challenging to follow up with those at high risk. D. Is the Link Workers Program Making Progress Towards Achieving its Intended Goals, Objectives and Outcomes? What Have Been the Challenges? AVERT End of Project Evaluation xi The LWS, funded by the Global Fund with USAID only funding management support, is thinly spread, reaching relatively small numbers of high-risk people in rural areas. LWS has reached 94% of the mapped high-risk populations, but there are questions about the accuracy of the mapping. The program has registered 14,452 rural high-risk persons above those already known to TI programs, and a further population of 496,541 vulnerable people. It has also identified a further 6,973 PLHIV. Many of those identified by the LWS are rural FSWs. One challenge the program faces is low salaries that do not attract males to the program; 70 - 85% of link workers are young females. Referral and follow up is difficult for those identified in need as services are in urban areas. 3) Capacity & scale-up of targeted intervention programs: A. Assess the Extent NGO Capacity to Carry out Core Skills in Targeted Interventions has been Fully Developed. With the exception of one NGO providing OVC services, all the NGOs and CBOs visited had received training to assist them in carrying out core TI skills. They had also had their MIS strengthened and had training and coaching in MIS. Unfortunately, because salaries are low, there is a high turnover of outreach workers (ORWs) and many current ORWs are not trained. The evaluation found that many ORWs had misunderstandings about basics; PEs and VPLs often appear to have better understanding but there appeared to be a further knowledge gap with the beneficiaries. None of the NGOs had QA systems for monitoring the quality of the BCC provided by the volunteers. B. How Effective Was the Technical Support Unit (TSU) in Supporting SACS to Scale Up and Improve the Quality of the TI Program? TSUs provided additional capacity to the SACS to scale-up TI interventions—from 44 (2007-08) to 96 (2010-11) in Maharashtra, and from 10 to 19 in Goa over the same period, with particular improvement among migrants and truckers interventions. It has also fostered more accurate data reporting with a focus on the actual target populations. The TSI has also facilitated a smooth transition of TI programs from other partners as their support from the SACS came to an end. The TSUs need to develop a set of quality indicators for their TI program interventions—particularly the BCC component—over and above the NACO NGO grading tool. 4) Assess the effectiveness of the Avert society various management systems: A. Is there a system in place that assists the staff in capturing, managing and analyzing program data? Avert has an extensive system in place that assists staff in capturing, managing and analyzing program data. B.Is there a systematic process for ensuring data quality control at all levels of implementation, including spot checks? There is a systematic process in place for data quality assurance (DQA) and it is being used to ensure consistent and regular data quality control both at the NGO and Avert Society level. C. How effective is the Avert M&E system in tracking progress? Avert ’s M&E system is effective at tracking progress against a large number of outputs and process indicators as required by both NACO and PEPFAR. It is much less effective at monitoring and tracking qualitative aspects of the program. AVERT End of Project Evaluation xii D. What were the strengths and weaknesses of the Avert project related to governance structures? Strengths: The GB is well placed for monitoring Avert performance, and for sharing experience and lessons learned with the other AIDS societies in Maharashtra, and potentially with other state societies through NACO. Weaknesses: Requiring GB approval for management decisions at operational level means that project performance can be greatly hampered by the GB not meeting to give needed approvals. E.What were the strengths and weaknesses of the Avert management systems? Strengths : Robust systems; Avert phase II began with a Management Systems Manual; weekly team leader meetings; monthly whole staff meetings. Weaknesses: Need for updating the management systems and the operational manual; lack of management technology and software. Project Planning & Review: Annual Action Plans; Avert Status Documents in 2005 and 2008-09 include strategic plan but no PMP. Grants Management: rigorously implemented transparent process for selection of grantees; program supervision and quarterly review meetings; annual experience sharing and review meetings. Financial and Procurement Systems: replenishment of revolving fund from NACO slow and fund size inadequate. Sound basis for financial management and procurement, although there is some unnecessary bureaucracy. F.What were the strengths and weaknesses of the Avert leadership team in steering the project? Leadership Gaps: There were many changes in leadership over the life of the project, although the current Associate Director has been in post since 2007. Weaknesses: Leadership has not been able to strengthen gaps such as developing a results framework and PMP; defining good minimum clinical care packages for FSWs and MSM regular health checks, establishing service delivery quality standards, programming for QA/QI (although data quality assurance is in place), and monitoring quality of service delivery beyond the NACO NGO assessment format. G. How effective was the coordination between various partners including MSACS, MDACS, other donors and stakeholders in maximizing resources through complementary planning and avoidance of duplication of efforts? At the state level there is a system of “one district, one donor”. At the district level the establishment of DAPCUs potentially increases the opportunity for coordination and complementary planning. Deployment of Avert district managers supports complementary planning to maximize synergies between Avert TI programs. The best example of planning to maximize use of resources is the mobile ICTC/STI vans that serve all the TI programs in a district. H. Were the planning, management and coordination systems adequate to ensure coordination and synergies of all the HIV prevention efforts? What were the challenges faced in coordination? The TI programs and the LWS in a district cover different populations, with TI programs in urban areas and the LWS in more rural areas. District level NGO planning and coordination for prevention occurred as part of the monthly meetings with DAPCUs and with the DM. Challenges: Major challenges include the turnover of NGO staff; movement of migrants; changing sex work and MSM cruising using of cell phones and websites. DISCUSSION OF THE THREE OVERARCHING EVALUATION QUESTIONS AVERT End of Project Evaluation xiii The three overarching evaluation questions are discussed in Section 5 of this report with the conclusion that the Avert Society model has greatly limited the effectiveness of the Avert Project. Nonetheless, Avert staff have achieved project successes, making a difference in the priority districts of Nagpur and Aurangabad in the fight against HIV— without the performance management benefits of a PMP, and despite the lack of enablement from the governing board. Section 6 summarizes six lessons learned from the Avert Project. RECOMMENDATIONS 1. It is crucial that a plan for transitioning Avert programs and important functions to MSACS be agreed upon and that Avert Society receive the necessary authorizations from USAID through NACO to implement the transition plan over the remaining period of the Avert Project. 2. Future HIV/AIDS activities would be more effective if they are in the non￾governmental sector, without the limitations and restrictions of the government system that result from inflexible implementation of NACO policies, procedures and guidelines that too often place a cap on service standards, rather than define the minimum service delivery requirements. 3. Project governance should not vest approvals for expenditure and implementation decisions within the annual plan and budget, in an external body whose members are not accountable for project performance. Decision-making within the annual plan and budget are better vested in the project director and an operations or finance director who are accountable for project performance. They might be guided by a non-executive, technical advisory committee. Annual plans and budgets should be approved by USAID/India with the Government of India for bilateral projects. 4. Performance monitoring plans, based on the hierarchy of results for a project, are important for monitoring performance against the agreed strategic objective, indicators and targets. The hierarchy of results might be in the form of the results framework used for strategically planning the intervention, that contributes to USAID/India’s Mission results framework SO14; or, if used for designing the project, a Logframe. 5. Future HIV/AIDS initiatives should ensure that QA/QI systems are addressed during the project design phase so that they contribute to and support the Project Logframe, as well as the performance monitoring plans. Design of these systems should reflect a thorough review of QA/QI systems being utilized by other HIV/AIDS, BCC and broad￾based health projects in India as well as best practices world-wide. 6. Use of MIS for evidence-based decision making is more likely to be sustained after a project if NGOs are trained to do their own analysis of trends and programmatic challenges. During the remaining months of the Avert Project, Avert should train and coach its subgrantees in interpreting their MIS data and making evidence-based management decisions to improve their programs. 7. Technical assistance and training in development and monitoring of QA/QI should be included in any new HIV/AIDS assistance along with M&E indicators that measure quality of services and activities. AVERT End of Project Evaluation xiv 8. The quality of BCC with program beneficiaries should be a future focus area, since reduction in risk-taking and improvement in health-seeking behavior is directly linked to the knowledge levels of the beneficiaries. Consistent with recommendation 5 above, development, implementation and monitoring against QA/QI systems for BCC should be addressed during the project log-frame and PMP development during the project design process. 9. NACP Phase Four should adopt (i) the TSU, as this is an effective approach; and (ii) the mobile ICTC strategy which Avert has demonstrated to be an effective use of resources to bring HIV/STI services to hard to reach vulnerable populations and to increase the uptake of HIV services. 10. The Link Worker Scheme should be reviewed since it is spread thinly, and since best practice for reaching MARPS is through peer to peer strategies. Many of the link workers in Maharashtra are reported to be young women, as the salary levels are too low to attract men, and none seem to be peers of the targeted populations. AVERT End of Project Evaluation 1 I. INTRODUCTION India has the third largest HIV/AIDS burden in the world. Although adult HIV prevalence at national level has declined from 0.41% in 2000 to 0.31% in 20091 , India’s population size is second only to China. Even a small increase in the HIV prevalence in India would be a human tragedy and would become an enormous drain on the national economy. It could also have global ramifications. Furthermore, the national figure masks the more complex variation in state and district-level prevalence throughout the south and northeast of the country. India bears multiple, concentrated epidemics with the highest prevalence among most-at-risk populations (MARPs): injecting drug users (IDUs) and their sexual partners, commercial sex workers (CSWs) and their clients, and men who have sex with men (MSM). There is also a growing concern that interstate migration in India could be fueling the HIV epidemic. The state of Maharashtra has a population of over 96 million. It is one of the six Indian States in which HIV spread into the general population with an adult prevalence of 1.2% in 2002. This decreased to approximately 0.25% for the rural population and 0.75% for urban areas in 2008. Nonetheless, the 2008 HIV Sentinel Surveillance (HSS) data indicates that prevalence remains greater than 1.0% in Mumbai and 4 districts of Maharashtra. Overall, antenatal prevalence in the state, at 0.63%, is still above the national average of 0.31%, though there has been a decline among pregnant women attending antenatal clinics from 0.88% in 20042 . Factors that contribute to Maharashtra’s vulnerability to the HIV epidemic include the sex industry in well-recognized areas of the state capital, Mumbai, and several other districts. There is extensive migration to and from neighboring states that have well￾established and growing HIV epidemics (Karnataka and Andhra Pradesh), and there are major transportation routes connecting Maharashtra to them. Maharashtra, India's leading industrial state, and Mumbai, India’s financial capital and largest city, are major destinations for migrants from various states of India. The Government of India (GoI) is now implementing the third phase of its $2.5 billion National AIDS Control Program (NACP-III), 2007-2012. The National AIDS Control Organization (NACO), a division of the Ministry of Health and Family Welfare, provides leadership to the NACP through 35 State and Municipal AIDS Control Societies (SACS) including Mumbai District AIDS Control Society (MDACS). Within states, implementation of the NACP is coordinated by District AIDS Prevention & Control Units (DAPCUs).3 USAID and the Bill and Melinda Gates Foundation (BMGF) are the two major donors funding programs that complement the efforts of Maharashtra SACS (MSACS) in scaling-up HIV prevention, care and treatment programs. BMGF’s primary focus was supporting prevention programs among MARPs in 13 high-prevalence districts in Maharashtra. BMGF is currently transitioning its programs to MSACS as it approaches the 2012 end of its current agreement. Additionally, UNICEF provides technical assistance on the prevention of mother-to-child-transmission (PMTCT). The Clinton Foundation is supporting pediatric antiretroviral treatment (ART) services. The state also receives funds from the Global Fund to fight AIDS, Tuberculosis, and Malaria (GFATM) for scaling-up integrated counseling and testing and care and treatment (ICTC) programs. AVERT End of Project Evaluation 2 II. BACKGROUND AND PURPOSE USAID has supported the Government of India’s effort to reduce HIV prevalence and mitigate the impact of AIDS in the country for over two decades. During US Fiscal Year 2010, funding in excess of $22 million focused primarily on programs in four priority states of which Maharashtra is one. USAID support to Maharashtra is through the bilateral Avert Project, signed in 1999 between the Government of the United States (USG) and the GoI to reduce the impact of HIV/AIDS in Maharashtra. The Avert Project —“Avert”—supports implementation of NACP-III and works in collaboration with the MSACS. AVERT PHASE I The first phase of Avert was launched in 2001 and ended in 2006. Project strategies included prevention, care, treatment, communication, research and capacity building. These strategies were implemented by the Avert Society, which supported NGOs, CBOs, networks of people living with HIV (PLHIV) and other institutions. Support was given to Johns Hopkins University and the Hindustan Latex Family Planning Promotion Trust for the design of behavior change communication (BCC) and condom social marketing programs for MARPs. In the first phase, Avert supported over 74 NGO projects to implement targeted interventions among MARPs, workplace intervention programs, care and treatment and capacity building of providers. An external evaluation of Avert was carried out in November 2005. It concluded that Avert was having “significant impact at the district and community level”, highlighting the successful establishment of the Avert Society with committed, well-qualified technical staff and the development of a cadre of effective peer educators. Technical areas for improvement noted by the evaluation included weakness in mapping efforts, a need to re-orient the focus of information education and communication (IEC) toward behavior change and to strengthen referral linkages for sexually transmitted infection (STI) diagnosis and treatment. Management areas for improvement included high staff turn￾over rates and weak senior management, slow disbursement of funds, a need for continuous monitoring to avoid duplication of effort among the Avert Society, MSACS and MDACS and improvement in responsiveness of the governing board (GB) were also documented. The evaluation further recommended development of key indicators to assess quality of services delivered, reach and effectiveness of the program and program sustainability. A no-cost extension of the Avert Project was recommended. AVERT PHASE II The second phase of Avert was approved for the period October 2006 to June 2011. The goals for this period were to i.) scale up prevention activities to support saturated coverage (85-90%) of MARPs in five high-prevalence districts; ii.) demonstrate models in community mobilization activities to increase the uptake of various care and treatment services in five districts; and, iii.) scale up work-place interventions in the entire state. In February 2009, a management review of Avert highlighted several weaknesses in the management system, including the procedure for recruiting senior positions. It recommended several key actions necessary to strengthen both the management and governance systems. AVERT End of Project Evaluation 3 CHANGES TO THE PROJECT Beginning in 2008, a significant number of additions and changes in focus were made to the Avert Project through NACO policy guidance and mandates that included implementation of: • Programs targeting high-risk migrants for targeted intervention (TI) activities • Technical Support Units (TSUs) in Maharashtra and Goa states to assist MSACS, MDACS and the Goa SACS to scale-up and improve the quality of TI activities • The Global Fund-supported Link Workers Scheme (LWS) throughout Maharashtra • The State Training Resource Center (STRC) for training all the NGOs in the state on core skills for the TI programs for MARPs THE EVALUATION PURPOSE AND OBJECTIVES USAID/India requested this end-of-project evaluation to assess and document the successes, failures and lessons learned in complementing the efforts of the Government of Maharashtra (GoM) to reduce HIV prevalence in the state. The objectives of this evaluation are to: 1) Assess the effectiveness of the Avert Project in terms of process, outcomes and impact: a) Saturating coverage of MARPs b) Improving quality of services c) Increasing consistent condom usage d) Reducing HIV prevalence 2) Assess the contributions of the Avert Project in developing HIV prevention programs for migrant populations; 3) Assess the effectiveness of the various management systems such as grants management, finance, and monitoring and evaluation in implementing prevention, care and treatment programs; 4) Document the programmatic challenges and lessons learned in implementing the Avert Project; 5) Make suitable evidence-based recommendations for the future directions of USAID HIV/AIDS support in the state of Maharashtra. The Statement of Work for the evaluation, including the final revised Evaluation Scope of Work, with overarching questions and detailed evaluation questions is included in Annex A. The evaluation team agreed upon a restructuring of the Evaluation Scope of Work with USAID/India during the in-country team planning meeting (TPM). This modified the presentation and order of the evaluation questions, but not the overall content of the Scope of Work. AVERT End of Project Evaluation 4 MAP OF MAHARASHTRA Figure 1 Map of Maharashtra showing Avert Districts at the time of the Evaluation AVERT End of Project Evaluation 5 III. EVALUATION APPROACH AND METHODS THE EVALUATION TEAM Two international evaluators and two national evaluators were contracted by Social Impact and its partner MSI to undertake the evaluation: Dr Ruth Hope, Team Leader, Senior Technical HIV/AIDS Expert and HIV/AIDS Analyst, Dr Deepak Khismatrao, Public Health Specialist and HIV/AIDS Analyst, Mr Keerti Pradhan, Senior Public Health Specialist/Management Expert, and Ms Barbara Spaid, Senior Public Health Specialist/Evaluation Methods Expert. EVALUATION APPROACH The evaluation was conducted through review and analysis of project and relevant Maharashtra health sector documentation. These included project agreements and reviews, Avert monitoring and evaluation (M&E) data—much of it raw—papers drafted for peer review, and other Maharashtra-specific HIV literature. The team also conducted program visits, interviews and beneficiary group discussions, with the intention of achieving broad and representative involvement of as many Avert stakeholders as possible. To this end, key informant interviews (KII) were undertaken with Government of India officers at national, state and district levels; a USAID officer, Avert Society staff members, nongovernmental organization (NGO) and community-based organization (CBO) staff and outreach workers, networks of PLHIV, and other stakeholders at project and community levels. Group discussions were conducted with community volunteers—Peer Educators (PEs) and Voluntary Peer Leaders (VPLs)—and beneficiaries. Annex B provides a list of all the persons contacted for the evaluation EVALUATION METHODOLOGY Three methods of data collection, triangulation and verification were employed during the evaluation. These were: 1. Review of Avert, USAID/INDIA, PEPFAR and other relevant documentation. 2. Key informant interviews with a wide range of Avert staff members and other stakeholders. 3. Group discussions with project beneficiaries. To maximize the number of programs visited during the field activities, the team split into two gender-balanced sub-teams, each with an international and a national evaluator. The field data collection was conducted through a subjective, qualitative examination of program implementation. The approach and tools were designed to achieve maximum triangulation of findings within the limitations of the assignment. While both field teams obtained as much in-depth information as was possible in the time available, it was neither possible nor appropriate to explore subjects in great detail. This report does not represent a quantitative evaluation although comparative quantitative impact data was obtained from relevant Behavioral Surveillance Surveys (BSSs). The evaluators developed specific tools, including key informant interview schedules for the various stakeholders, and group discussion guides for volunteers and beneficiaries. These tools are included in Annex C. Each of the two sub-teams took notes during the interviews and discussions and later typed them up and shared them with all four team members. After the field data was collected, the team worked together in Delhi to discuss and analyze the findings in relation to each of the evaluation questions. This ensured the findings from both sub-teams were reflected in the analysis by the writers of the findings section of this report. AVERT End of Project Evaluation 6 EVALUATION SCHEDULE Please see Annex D for the evaluation calendar. The evaluation began in March 2011 with a review of advance documentation made available by USAID/India and a Team Planning Meeting (TPM) for the international evaluators in Arlington Virginia. The international evaluators traveled to India at the beginning of April 2011 and held further TPM days with Dr Khismatrao and USAID/India, as well as developing the Key Informant Interview (KII) schedules and conducting the first KII with the Project Management Specialist/Activity Manager of the Avert Project Health Office, USAID/India. The team traveled to Mumbai for 2 days for briefings and KIIs with Avert staff, which led to a reorganization of the planned field visit schedule to maximize the number of programs visited in the time available. The team developed a Group Discussion guide in Mumbai. Mr Pradhan joined the team at the beginning of the second week of April. The Team Leader briefed him on the plans and evaluation activity to date before he participated in the field visits. The team returned to Mumbai for a mid-evaluation debriefing with USAID/India between the first and second week of field visits. During the fourth week of April 2011, the team worked together in Delhi analyzing findings and preparing for the final debrief with USAID/India which was held on May 2, immediately before the international evaluators returned to the USA. The final report was submitted through Social Impact to USAID on June 6, 2011. The team visited 27 of Avert’s current 41 NGO/CBO interventions and 4 Link Worker Schemes (LWS) in Avert’s six districts: Nagpur, Aurangabad, Jalna, Thane, Solapur and Satara. Table 1 Summary of the Avert Interventions and those visited in each district by type of intervention Type of Interve ntion NAGPU R AURANGBAD JALN A THAN E SOLAPU R SATAR A Aver t Total Visited Aver t Total Visited Aver t Total Visited Aver t Total Visited Aver t Total Visited Aver t Total Visited FSW 4 3 3 2 1 1 1 - - - - - MSM 1 1 1 1 - - 5 1 - - - - IDU - - - - - - 1 1 - - - - Migrant 5 4 3 1 1 1 7 3 3 2 - - Trucker 1 - - - - - - - - - - - DIC 1 1 - - - - 1 1 - - 1 1 OVC - - - - - - 1 1 - - - - WPI - - - - - - 1 - - - 2 1 Total: 12 10 7 4 2 2 17 7 3 2 3 2 Key: FSW female sex worker DIC drop in center IDU injecting drug user MSM men who have sex with men OVC orphans and other vulnerable children WPI workplace initiative AVERT End of Project Evaluation 7 IV. FINDINGS 1) ASSESSMENT OF THE EFFECTIVENESS OF THE AVERT PROJECT IN TERMS OF PROCESS, OUTCOMES AND IMPACT IN: A. Saturating Coverage of MARPs Through Scaling-up of Targeted Interventions The original Strategic Objective from the 2001 Avert Tripartite Agreement was to “increase the use of effective and sustainable responses to reduce the transmission and mitigate the impact of STI, HIV and related infectious diseases”. A major focus was the commercial sex industry. The second phase of the Avert Project in 2006 reflected the latest NACO and PEPFAR guidance and was informed by a newer and evolving understanding of effective and sustainable methods for reducing transmission and mitigating the impact of STIs and HIV in Maharashtra. Under phase two, the strategic objectives focused on saturating coverage (85-90%) of MARPs, as well as on high-risk migrants, demonstration of comprehensive models for care and support, and community mobilization in five high-prevalence districts. To date, Avert has almost achieved its target for 85-90% saturated coverage. Against FY 2010 PEPFAR indicators (P8.1.D, P8.2.D and P8.3.D) that measure targeted populations reached through individual and small group level interventions, Avert has reached 76% coverage as of September 30, 2010. Against NACO indicators that measure regular contact against active population, as of December 2010, Avert has achieved 72% saturated coverage of FSWs; 79% of MSM and 76% of single male migrants with high￾risk characteristics. Despite slight differences in measuring and defining saturation of coverage, the achievement of saturated coverage measured by both PEPFAR and NACO is remarkably similar and close to target. This saturated coverage of high-risk populations, as well as numerous additional outcomes under additional priority areas, has been achieved by Avert through financial and technical support to over 90 NGOs, CBOs and networks of PLHIV that are implementing prevention, care and support, and treatment activities in the priority districts of Maharashtra. B. Increasing Consistent Condom Usage Among MARPs The 2009 Behavioral Surveillance Survey (BSS) for Maharashtra, carried out under the Avert Project, estimates that 80% of HIV transmission in the state is due to unprotected sex. This makes correct and consistent condom use (CCCU) a key behavior change necessary for effective prevention of STI and HIV transmission among commercial sex workers. Behavioral indicators tracked by four separate BSSs since 2004 reflect fairly consistent Difference in CCCU between paying and non-paying partners, Nagpur BB-FSWs 64 77 91 95.6 3 19 35 56.1 0 20 40 60 80 100 120 2004-2005 2005-2006 2007-2008 2009 Year BB-FSWs CCCU with paying partners BB-FSWs CCCU with non-paying partners AVERT End of Project Evaluation 8 improvement in CCCU among both brothel-based (BB) and non-brothel-based (NBB) female sex workers (FSWs), as well as among men who have sex with men (MSM) in those Avert priority districts that were included in the BSS sample. There is, however, considerable difference in CCCU between paying and non-paying partners among both FSWs and MSM. For example, in Nagpur, CCCU among BB-FSWs has increased from 64% in 2004 to 95.6% in 2009 with paying partners; but only 56.1% BB-FSWs reported using condoms consistently with non-paying partners in 2009. There is less of a gap in CCCU between paying and non-paying clients among MSM; however, overall CCCU is lower in MSM in general than among FSWs. For example, in Thane, CCCU among MSM has increased from 69% in 2004 to only 73% in 2009; but 66.5% of MSM report CCCU with non-paying partners in 2009. More extensive findings for CCCU in Avert priority districts can be found in Annex E. Many peer educators and outreach workers contacted in the field attribute this increased CCCU among FSWs and MSM to their improved understanding of the risks associated with unprotected sex., which they credit to BCC skills training provided under the Avert Project. During group discussions with FSWs, several cited training in condom negotiation skills and availability of female condoms as contributing factors to improved CCCU. On the other hand, all too many peer educators and outreach workers, when asked about consistent condom use in their project areas, proceeded to describe it in terms of numbers of condoms distributed. This probably reflects the considerably greater emphasis placed on reporting condoms distributed than on CCCU. As one senior government official noted, there needs to be greater emphasis on CCCU rather than on condoms distributed if Maharashtra is to further decrease its HIV prevalence. In response to concerns raised by the evaluation team about the reliability of self￾reported CCCU, both to BSS interviewers as well as to peer educators, several peer educators admitted they feel certain sex workers over-report consistent condom use. They say they are certain of this because they know their community members well. Also, while the current outreach to migrants is a relatively new focus under Avert, there is some evidence based on the evaluation field visits that not all migrants are routinely being advised about the risks of unprotected sex and the importance of using condoms correctly and consistently. While there is considerable room for programmatic improvement in CCCU, much credit for increasing CCCU in Avert districts over the past decade must be given to Avert. C. Improving the Quality of Services USAID’s 2006 Project Approval Document for phase two of the Avert Project placed little emphasis on quality assurance, or on improving the quality of services beyond a brief mention under project priorities to “strengthen monitoring and evaluation activities to improve the quality of interventions”. Avert is monitored and evaluated against NACO and PEPFAR reporting indicators that are primarily quantitative in nature. Thus, in the absence of a Project Monitoring Plan (PMP) with M&E indicators to track quality of services, or a written directive from USAID to the Avert Society to ensure that quality assurance (QA) should be incorporated into the program, it is not surprising that there was not a visible focus or concerted effort on QA and quality improvement (QI). This is not to say that services encountered and observed in the field were necessarily of a poor quality, but that the quality of interventions was inconsistently and unsystematically monitored. In the field, NGO Project Coordinators were asked to describe their QA system for service delivery and were consistently unable to do so. When asked how they monitor the quality of their services and activities, their response was generally to describe their procedures for ensuring the quality of their data collection and reporting. AVERT End of Project Evaluation 9 Too often when asked how they know the BCC is effective, they stated that the IEC materials were developed and pretested by NACO and so they must be effective. Because effective BCC is critical to the success of targeted interventions among MARPs, the evaluation team focused specifically on the quality of BCC messages being provided by outreach workers (ORWs), PEs and VPLs, and to routine monitoring of their BCC in the communities they serve. Group discussions held with beneficiaries (migrants, IDUs, FSWs and MSM) highlighted a fairly good but still imperfect level of understanding of key BCC messages regarding HIV transmission, the risks of unprotected sex, and the optimal frequency for STI and HIV testing. There were notable misconceptions: one FSW said she has her blood tested monthly; a transgendered person explained that she was protected from STIs by taking syndromic treatment drugs every six months; a migrant worker claimed that HIV is transmitted by mosquitoes; an MSM understood that routine STI tests are not necessary if an HIV test is negative. Questions the evaluators directed to PEs and ORWs regarding specific aspects of BCC messages were generally answered correctly, but there were sufficient incorrect responses that show less than perfect command of BCC messages. When asked specifically how BCC message content conveyed by ORWs, PEs and VPLs was being assessed and monitored in the field, it was evident that a few NGOs had systems in place, but most did not. It was clear that there was no standard QA/QI system for service delivery under Avert. There were scattered examples of QA monitoring happening at the NGOs but it was being done inconsistently and on an ad hoc basis. For example, one project coordinator had been trained in QA by another donor agency and said he spent four days each month in the field monitoring the work of the ORWs; some ORWs mentioned holding group discussions with migrants or FSWs to cross check messages being given by the PEs. Several PE supervisors also said that they provide monthly refresher training to PEs on key messages such as STI symptoms, HIV transmission and correct condom use. However, there were no QA standards or checklists being used to monitor PE/VPL performance. Several project coordinators, when asked, said they did not focus on or monitor the quality content of the information being shared by their ORWs, PEs, VPLs or Link Workers because their IEC material was developed and pretested by NACO. One NGO project coordinator reported that she had requested technical assistance from Avert to develop quality standards and a quality assurance monitoring system but the assistance had not yet been provided. D. Reducing HIV Prevalence At this point, the data on HIV prevalence are too insufficient and inconclusive to allow for a clear determination as to whether or not Avert has had an impact on lowering HIV prevalence in Avert priority districts. Ideally, one would want to see steady decline in HIV prevalence among FSWs, MSM or at STI sites over a number of years in project districts to be able to comfortably make that attribution. Unfortunately, the HIV Sentinel Surveillance (HSS) carried out annually in Maharashtra did not sample FSW prevalence in Avert districts over a span of years except in Thane, where prevalence was 38.0 in 2004, 25.2 in 2005, 28.4 in 2007 and back up to 32.4 in 2008 for an overall U-shaped decline between 2004 and 2008 of 17.0%. Nagpur and Aurangabad, two key Avert priority districts not included in HSS sampling prior to 2008, show HIV prevalence of 17.2 and 2.0 respectively in 2008. In Thane District, Avert is implementing a targeted intervention program with MSM and high-risk migrant populations, while BMGF is implementing a targeted intervention program among FSWs. AVERT End of Project Evaluation 10 Data available from STI sites show a decline of 36% in Nagpur from 21.2 in 2002 to 13.6 in 2008, but a 31% increase in Thane from 8.3 in 2002 to 12.0 in 2008. HIV prevalence from antenatal clinic (ANC) populations in Avert priority districts shows considerable upward and downward movement. There is, however, eventual decline between 2002 to 2008 in Aurangabad. Prevalance fell from 1.25 to 0.25 in 2008. In Nagpur it fell from 2.25 to 0.63 and in Thane from 2.0 to 0.75 over the same six-year time span. (Annex F) A strong downturn in HIV prevalence among FSWs, MSM and STI sites in Avert priority districts in the 2010 and 2012 HSSs support to the argument that Avert has indeed had a major influence on HIV prevalence. Also, the fairly recent and strong uptake in outreach and service delivery occasioned by Avert’s ICTC mobile vans, strengthened PE efforts, the Migrant Workers Program and LWS would not be reflected in HSS data before 2010. E. How successful were the efforts of the Avert Project in developing community￾based organizations and empowering MARPs to access services? Four of the seven CBOs supported by Avert were visited by the evaluation team: Prerana Samajik Sanstha (PSS) and Udaan in Aurangabad; Sarathi in Nagpur; and Humsaya in Thane. PSS is a CBO composed of FSWs and the other three are composed of members from MSM and transgendered communities. All four CBOs have received support from Avert for at least the past two years and two had received support for four years. Members from all four CBOs said that significant benefits had accrued to their community members from the advocacy work they carried out through “pressure groups” with Avert assistance, including reduced stigma and discrimination, improved rapport with government officials, decreased police harassment and more respectful treatment at ICTC and antiretroviral therapy (ART) centers. In addition, the CBO members mentioned the importance of now having a platform or mechanism for sharing information and supporting each other, as well as improved crisis management, increased access to social entitlements and increased self-respect and confidence. Many CBO members, particularly at PSS, have begun to access social entitlements they were previously unaware of, including ration cards, payments for widows and abandoned women, school fee scholarships for their children, and life insurance. All four CBOs have formed self-help groups (SHGs), which have linked members to income generating activities (IGA) and credit facilities, empowering them economically and increasing their self-respect and self-reliance. One SHG formed by transgendered members of Sarathi is in the process of registering as a CBO – graduating from Sarathi HIV Prevalence at STI Sites 0 5 10 15 20 25 2002 2003 2004 2005 2006 2007 2008 Year Aurangabad Nagpur Thane AVERT End of Project Evaluation 11 support now that they have increased self-esteem and confidence. Clearly, community members have been empowered through CBO formation, advocacy efforts and membership in SHGs. In addition, improved service uptake in 2010-11 occurred in all four CBOs, particularly in increased HIV testing. (See Table 2). Presumably the direct involvement of community members in running the organization, and the increased empowerment of members, contributed to improved access and utilization of services, but there is no direct evidence to support this. At least nine (40%) of the NGOs visited by the evaluation team described similar successes in community mobilization and empowerment through access to social entitlements; the development of education, health and crisis committees for improved support; linkages to loans and IGAs through SHGs and increased service uptake for 2010-11. It appears that Avert’s innovative and holistic approach to mobilizing and supporting the NGOs and CBOs has proven to be the tipping point for successful community mobilization and empowerment, rather than simply the formation of CBOs. TABLE 2: Community Mobilization Characteristics of Avert CBOs Prerana Samajik Sanstha (PSS) Udaan Sarathi Humsaya Supported by Avert since 2008 2009 2007 2007 Results of Advocacy Efforts Reduced S&D; improved rapport with govt. officials; less police harassment Reduced S&D; improved treatment at ART Center; less harassment by police Increased confidence to speak out on MSM issues; Reduced S&D; better treatment by ICTC & ART Center staff Benefits to Community Assistance in accessing social entitlements; increased respect due to improved knowledge Gave MSM means to share info and support each other DIC next door to a community hospital has increased access to services; practicing safer sex Improved crisis mgmt. SHGs 16 SHGs 1 SHG 2 SHGs One about to graduate as a registered CBO 8 SHGs Empowerment Women linked to income generating activities (IGAs); Increased employment opportunities – Sarathi Director emphasized the importance of economic opportunities to keep MSM out of sex work Service Uptake in 2010-2011 from previous year Increase in HIV testing from 1238 to 1449 Increase in HIV testing from 148 to 910 Increase in HIV testing from 481 to 1124 Increase in HIV testing from 365 to 1246 Key: S&D stigma and discrimination AVERT End of Project Evaluation 12 F. How Effective Was the Avert Project in Developing a Strong Peer Education Program to Deliver Prevention and Care Services at The Community Level? Avert, with the Belgaum Integrated Rural Development Society, developed and piloted the voluntary peer educator model in 2004 to help prevent HIV and STI infection among core groups and bridge populations. The PE model was later sanctioned by NACO and is not only an important strategy for Avert, but for the national program as well. In 2009, NACO determined that all PEs were to receive a monthly stipend of Rs1500 along with a Rs500 travel allowance in an effort to standardize the national TI strategy. As of March 2011, there were 388 active peer educators (PEs) working with FSW, MSM and IDU communities under the Avert Project. Of these, 303 have been trained. There are also 1710 voluntary peer leaders (VPLs) working with migrant populations, of whom only 302 have been trained. The PEs work within their communities, providing information on HIV and STI transmission, condom use, and safe sex practices; mobilizing the communities for service uptake; and creating an enabling environment for empowerment of community members. The evaluation team met with more than 50 PEs and VPLs, both in their communities and at the NGO/CBO offices. On several occasions when both PEs and ORWs were present, evaluators noted that the PEs were more knowledgeable, dynamic and motivated than the more highly-educated and better paid ORWs. Clearly, the PEs are playing a critical role by connecting their communities to information and services. This was confirmed by NGO project coordinators; Avert District Managers, DAPCU officials; and ART center staff. In group discussions conducted by the team in the field, PEs mentioned that they increasingly see themselves as peer role models so they take better care of themselves and have improved their personal hygiene. They also feel that they receive more respect in their communities because they are paid the Rs1500 honorarium and feel it recognizes the importance of their work. More confident, empowered PEs provide strong impetus toward empowering others in their communities. In late 2009, NACO instituted a new PE tracking sheet format and shortly thereafter, in early 2010, Avert organized training/retraining of PEs through the STRC. This retraining focused primarily on how to use the new PE tracking sheets and how to improve BCC outcomes through use of newly revised IEC materials. The evaluation team reviewed information from four NGOs visited and found that, following the release of the new PE tracking forms and the ensuing training/retraining of PEs in BCC and data collection, there were significant increases between June 2010 and February 2011 in the number of PE contacts, follow-up visits, one-to-group interactions, IEC materials distributed, condoms distributed and condom demonstrations, and re-demonstrations (see Annex G). It is also worth noting that the Avert Project’s district-wide tracking also shows a general increase in service uptake for 2010-11, particularly for HIV testing, which coincides with the increased activity by the PEs/VPLs as well as the roll-out of the highly successful mobile ICTC vans. A major concern, as noted earlier, is the lack of a consistent and articulated QA/QI system for monitoring- and improving on an on-going basis- the quality of PE/VPL activities including BCC. G. Has the Avert Project Made a Difference in Priority Districts, Especially Aurangabad and Nagpur? Avert has provided major support over the past ten years in reducing the impact of HIV and AIDS in the state of Maharashtra, primarily in scaling-up prevention activities, but also in care, support and treatment in priority districts. Two of these priority districts, AVERT End of Project Evaluation 13 Aurangabad and Nagpur, received their only support from Avert. It has been conjectured that much might be learned from a close look at how these two districts have performed in increasing access to and utilization of services, HIV prevalence figures, and key behavior changes. In matching these two districts to “companion districts” that share similar social and geographical characteristics and share administrative divisions, the team considered whether or not it is possible to say that the Avert Project produced greater results than were achieved in other, non-Avert supported, districts in Maharashtra (although it must be noted that Avert supports MSACS efforts in other districts through the TSU, LWS and STRC). For this purpose, Aurangabad’s performance was compared to that of Parbhani from the Aurangabad Division and Nagpur’s performance was compared to that of Chandrapur, from Nagpur Division. Parbhani was under the BMGF program from 2004-2010, when it was transferred to MSACS. Figure 2. Comparision of Avert (Aurangabad) vs. non-Avert Division’s (Parbhani) Trends in ICTC Testing Figure 3. Comparision of Avert (Nagpur) vs. non-Avert Division’s (Chandrapur) Trends in HIV Testing As seen in Figure 2 above, a comparison of trends between Aurangabad and Parbhani districts in ICTC testing, positivity rate, ANC prevalence and behavior change—as reflected by correct and consistent condom use (CCCU)—shows considerable similarity of performance between the two districts. While the overall number of HIV tests performed in Aurangabad by 2010 was higher than in Parbhani (46,026 compared to 31,184), the percentage of increase between 2007 and 2010 for Aurangabad was only AVERT End of Project Evaluation 14 28%, compared to a 102% increase in Parbhani. In Aurangabad the number of HIV tests actually dropped between 2009 and 2010. This is possibly a reflection of the recurring shortage of HIV test kits. The trend in the positivity rates between Aurangabad and Parbhani reflect a nearly identical decline: from 6.79 in 2007 to 3.26 in 2010 in Aurangabad for a 52% decrease; and from 7.08 in 2007 to 3.58 in 2010 in Parbhani for a 49% decrease. Trends in HIV prevalence among ANC clients show a small rise in Nagpur from 0.50 in 2004 to 0.63 in 2008, and no change from the 0.25 prevalence over the trend years. Ups and downs between 2004 and 2008 in HIV prevalence among ANC clients are fairly closely mirrored over the years. Behavior change as reflected in CCCU figures from the BSSs show strong increase for Nagpur from 46% CCCU among FSWs with paying partners in 2004 to 88.25% in 2009. There is no similar trend of figures from BSS for Parbhani, but the comparable figure for 2009 is 77.95. Again, this is a remarkably similar result. However, CCCU among FSWs with non-paying partners in Parbhani is almost twice as high in 2009 at 35.6 % as it is for Aurangabad at 18.1%. A similar comparison of trends between Nagpur and Chandrapur (see Figure 3 above) shows again very similar performance progress. The total number of HIV tests performed in 2010 fell for both districts, although the drop was more pronounced in Nagpur. The positivity rate fell in both Nagpur and Chandrapur from 2007 to 2010, by 60% and 50% respectively. HIV prevalence among ANC clients rose in both districts from 2004 to 2005 and fell by 2008 from 3.0 in 2004 in Nagpur to 1.51 in 2008, and from 1.25 in Chandrapur in 2004 to 0.63 in 2008. CCCU in Nagpur and Chandrapur for 2009 is very similar for FSWs with paying partners at 95.6% and 98.7% respectively. However, in Nagpur almost twice as many FSWs report CCCU with non-paying partners as in Chandrapur, 56.1% and 31.1 respectively. Complete data sets for both of these comparisons can be found in Annex H. It would appear that performance among Avert districts is positive and similar to that taking place in non-Avert districts where other donors and MSACS provide similar support. Clearly the Avert Project has made a difference in the Avert priority districts of Aurangabad, Nagpur, Jalna, Solapur and Thane in terms of expanded coverage of MARPs and increased uptake of services, particularly HIV testing by 2010-11. This is due largely, as noted earlier, to improved tracking systems introduced by NACO, the strong network of peer educators trained and supported by Avert, and the successful mobile ICTC vans introduced in 2010 by Avert. Qualitative data from the field also strongly supports this finding. NGO and DAPCU staff, as well as beneficiaries, confirm that the PEs and VPLs play a critical role in providing the FSW, MSM, IDU communities and the high-risk migrants with BCC and linking them with STI and HIV referrals, testing services and ART centers as needed. Both DAPCU and ART center doctors confirmed that the NGOs and PEs play a critical role in supporting HIV-positive persons and linking them to services. The holistic approach that Avert has taken in mobilizing and empowering communities through SHGs and community-based committees was mentioned and appreciated by both CBOs and NGOs repeatedly. Most organizations said that this approach has facilitated service uptake in their areas. AVERT End of Project Evaluation 15 2) ASSESS THE SUCCESS OF THE PROGRAM INTERVENTIONS AND MODELS FOR PREVENTION AND CARE DEVELOPED BY THE AVERT PROJECT: A. Challenges to District Comprehensive Prevention, Care Support and Treatment Programs in Scaling-up Access to Quality Prevention and Care Services by MARPS and Their Partners. Avert conceptualized a district comprehensive model encompassing holistic prevention, care and support, and treatment, and supports implementation through sub-grants to NGOs, CBOs and networks of PLHIV in its districts. Prevention activities with appropriate communication strategies are delivered through targeted interventions addressing the needs of MARPs—FSWs, MSM and IDUs, and high-risk male migrants. In some districts, Avert’s truckers TI and workplace initiative also deliver BCC related to STIs and HIV. Care and support services are provided by networks of PLHIV through support groups and DICs. Antiretroviral therapy (ART), when it became available, was delivered through government ART centers. Later, clients stable on ART were referred for follow up to Avert-supported community care centers (CCCs) although NACO then transferred responsibility for managing the CCCs to Karnataka Health Promotion Trust (KHPT). The LWS extends district prevention, testing and referral activities to rural areas where surveillance indicates an increased risk of transmission. Avert introduced health camps and mobile van’s providing ICTC and STI services to bring these services closer to hard-to-reach populations. Figure 6: Conceptual model for Avert’s District Comprehensive Program Key aspects of the implementation of Avert’s comprehensive model include referrals for HIV testing, STI diagnosis and syndromic management. The regular follow up of high￾Bridge Population Mapping Program/Services BSS HSS Core Groups General Population District Communication & Work Place Intervention Condom Promotion Plan M & E ICTC STI Health Prog. HBC OVC DIC ART DOTS Govt Schemes District LWS LWS LWS Capacity Building Key: DOTS directly observed therapy [for tuberculosis] HBC home-based care AVERT End of Project Evaluation 16 risk persons, and those who test positive for HIV, ensures access to a continuum of care, support and treatment. Sub-grantees receive capacity building in the form of management information system (MIS) strengthening, staff training related to HIV/AIDS/STIs, advocacy and BCC, and IEC and prevention materials. Close co￾ordination of all the services in a district, facilitated by the DAPCU, supported by Avert’s district manager (DM), maximizes saturation and avoids duplication and gaps. The DAPCU holds regular monthly meetings for co-ordination of all activities at district level. At least one DAPCU actively refers HIV-positive persons from ICTCs to NGO services and follows up to ensure they receive these services. Other issues addressed in the monthly meetings are stigma and discrimination, planning of HIV testing camps, tracing of persons lost to follow up, and planning of joint events such as advocacy campaigns in the week of World AIDS Day each year. The main challenges to Avert implementing its model have been repeated changes in NACO policies and direction. At the beginning of Avert Phase II, there was an emphasis on scaling-up workplace initiatives throughout Maharashtra. Later workplace initiatives were dropped from NACO priorities. Between 2002 and 2004 Avert was targeting slum populations and identified migrants living in slums as a high-risk population. However, Avert had to cease its slum intervention and there was then a gap before NACO guidelines included single male migrant interventions. Avert would like to include a TI program for prisoners, who are known to be at increased risk for HIV transmission, but this is not a NACO priority area. Loss of the CCCs was a blow to Avert’s program—and an even greater hole in Avert’s Integrated Care Model (see following section C). Avert identified the needs of HIV-positive children, and children affected by or orphaned by AIDS. The needs of these children are often not being met appropriately in its districts, but Avert is unable to establish subservices for this group as it is not part of the NACP￾III. Ideally, management of a large complex model for the continuum of care should have been coordinated through an electronic MIS with unique identifying numbers for clients that would permit tracking individuals, while maintaining privacy, through all the appropriate services over time. However, Avert was refused permission to purchase the necessary hardware and software systems as these were not in line with NACO guidelines. Instead, the MIS for the LWS does not adequately prevent repeated testing of the same low-risk individuals in rural areas, and positive clients from ICTC are recorded through a paper system that identifies them by name and yet does not have the power of an electronic system to track referrals and service use over time. Evaluation of the district comprehensive model is also a challenge for Avert. A complex model like this needs implementation within an operational research framework, with collection of monitoring data, against appropriate indicators for effectiveness from a baseline. This framework is not in place and baseline data was not collected. Furthermore, the repeated changes in program components in response to NACO directives have wasted investments in program components which were stopped before they were evaluated for effectiveness. This also had an impact on the morale and spirit of the Avert staff. Non-availability of STI drug kits until October 2010, and the frequently interrupted supply of HIV kits, were other important challenges in scaling-up services under the model. The transition of the district programs to MSACS under the coordination at district level, by the DAPCUs will offer Avert an opportunity to further strengthen coordination by AVERT End of Project Evaluation 17 supporting and coaching DAPCU staff to make visits to TI NGOs during the transition period. B. The Integrated Care Model Providing Care and Treatment Services to Adult and Children Infected and Affected by HIV/AIDS. What have been the Challenges to Integrating Adult and Child Care? Avert’s “integrated care model” is a subcomponent of its district comprehensive model, that aims to provide integrated care services for adults and children infected and affected by HIV/AIDS. Integrated care model services revolve around the DIC and support groups for PLHIV, with referral linkages to the ART center and CCC, and to social entitlements such as ration cards, ideally with peer support and counseling. The purpose of integrated care services is to maximize the quality of life for people infected and affected with HIV/AIDS. Discussions with DIC beneficiaries elicited their perceived benefits from attending the DIC which include peer support and understanding of difficulties faced by PLHIV, ability to speak with others about personal problems without being judged or subjected to stigma and discrimination; advice on social entitlements and other services available and guidance on how to access these, nutritional advice, adherence counseling and advice on how to manage side effects of ARTs – which the ART doctors are reported not to spend time providing. Some are able to join self-help groups which have small savings clubs and give small loans—which one beneficiary reported she has used to buy a sewing machine—or income generation activities. Figure 5: Conceptual framework for Avert’s integrated care model AVERT End of Project Evaluation 18 Avert-supported integrated care programs have put considerable effort into registering PLHIV, orphans and vulnerable children but have been challenged by the paucity of provisions for children affected by HIV/AIDS. In Satara the evaluators were taken to visit a remand home which has a wing for orphans and HIV-positive children. The president of the network of PLHIV and the staff of the remand home had poor understanding of the need for children to grow up within families. They considered it good for positive children to be sent to the remand home because the remand home provides schooling and is planning a trade school. Yet most of the positive children in the remand home have families in the district, as evidenced by only two being present on the day of the evaluation visit, which was on a holiday. The other residents had gone back to their villages to stay with grandparents and other extended family Another model for care and support of OVC is offered by Committed Communities Development Trust (CCDT), an NGO working with the children of FSWs in Thane. The CCDT model offers a mix of home-based care and institutional support for the children of FSWs in crisis situations. It is a holistic model, providing parenting, emotional, health, nutritional, educational, career guidance and vocational training services that promote the reintegration of the vulnerable child into their family in mainstream OVC services. Although CCDT has received financial support for service provision from Avert, the staff have not received any technical support from Avert and capacity-building workshops they attended did not seem relevant to their needs. They said there is a lack of coordination between the DIC and the needs of the OVCs. Another challenge for integrated care programs is meeting the needs of discordant couples and monitoring couples over time to record if they remain discordant—which would be an indicator of the effectiveness of services for discordant couples. Discordant couples are registered at the DIC, and the DAPCU also maintains a database of discordant couples, duplicating the effort. A challenge to long term sustainability of the DIC services is that many of the networks of PLHIV are collectives with elected officers serving for fixed terms rather than NGOs or CBOs with a Chairperson and other staff. This presents a real challenge for capacity building, with training being given to hired project staff running the DIC who are not living with HIV and simply operating under the name of the network of PLHIV. Yet Avert has been unable to evaluate an alternative model institutionalizing the DIC and support groups with the CCCs, as NACO transferred the CCCs from Avert technical assistance to KHPT. Because the DIC is centrally located at district headquarters, its services are really only available to PLHIV living in that part of the district. The model needs to be extended to facilitate support groups for PLHIV closer to where they live -- at taluka (neighbourhood) level throughout areas of the district where there are PLHIV living. C. HIV Prevention for Migrant Populations. What Have Been the Challenges? Avert is implementing a total of 19 HIV prevention TI projects for migrants in five districts—Nagpur (5), Thane (7), Aurangabad (3), Solapur (3) and Jalna (1). Of these, five projects in Nagpur and Thane districts received funding from Avert from 2002 through to 2004 for HIV/AIDS awareness programs in slum areas, reaching migrants living there. NACO then directed that the slum projects cease but from 2009 funding was instead provided for migrant interventions. All the migrant projects are NGO implemented with outreach workers and VPLs. AVERT End of Project Evaluation 19 Avert undertook a mapping exercise and situational needs assessment (SNA) for each of the migrant TI projects in December 2009, as a part of state-wide initiative to revalidate the high-risk and bridge population estimates. However, the estimated population of migrants may still not be accurate for Solapur, where Avert programs are reported to have achieved more than 200% coverage in 2009-10 and more than 150% coverage the following year. (See Table 3) In Thane, the district with the largest migrant population, the coverage is 21.3%, considerably lower than coverage rates achieved in the other districts. (see Table 4). Table 3 Avert Migrant Program Coverage in its Five Program Districts: District Migrant Estimates (Mapping exercise 2009) Migrants Coverage (Avert program Data) (Active Population) (2009 – 10) Migrants Coverage (Avert program Data) (Active Population) (2010 – 11) N % N % Nagpur 80,025 46,893 58.6 72,111 90.1 Aurangabad 37,842 19,869 52.5 30,815 81.4 Jalna 16,683 3,802 22.8 16,714 100.2 Thane 526,699 44,888 8.5 112,059 21.3 Solapur 15,174 33,467 220.6 22,840 150.5 TOTAL 676,423 148,919 22.0 254,539 37.6 Challenges Training of VPLs NGOs provide training for VPLs who provide the link between the NGO ORWs and the migrants. Not all VPLs are migrants. Some are canteen owners or shopkeepers who provide retail services to migrant workers. Many VPLs interviewed had good knowledge of HIV and AIDS and correctly demonstrated condom use. However, of the 1710 VPLs currently working with migrant programs, only 302 have been trained. All VPLs need training if they are to be effective, and to ensure uniform quality of VPLs across all the migrant interventions. Knowledge gap In beneficiary group discussions, the evaluation team found a clear gap between the VPLs’ level of knowledge and some of the beneficiaries who had misconceptions about HIV transmission and prevention. Coverage and Achievement of Targets Table 3 demonstrates that the number of migrants reached, the coverage, doubled between 2009-10 and 2010-11; simultaneously the service uptake for STI and HIV testing increased almost four times in absolute numbers. NGO staff, Avert District Managers and DAPCU officers attribute this to the mobile van instituted and the health camps conducted in the migrant interventions providing ICTC and STI checkups. AVERT End of Project Evaluation 20 Table 4 Progress on Key Indicators against increase in the coverage Districts Coverage STI Visits ICTC - HIV tests 2009-10 2010-11 2009-10 2010-11 2009-10 2010-11 Number (N) N N % N % N % N % Nagpur 46893 72111 4307 9.2 19918 27.6 1804 3.8 10210 14.2 Aurangabad 19869 30815 1981 10.0 7486 24.3 441 2.2 4258 13.8 Jalna 3802 16714 539 14.2 3608 21.6 219 5.8 3784 22.6 Thane 44888 11205 9 3486 7.8 18746 16.7 2125 4.7 9131 8.1 Solapur 33467 22840 4088 12.2 6698 29.3 2573 7.7 4578 20.0 TOTAL 148919 254539 14401 9.7 56456 22.2 7162 4.8 31961 12.6 Focusing on migrants practicing high-risk behaviors Few of the NGOs had undertaken individual risk assessments of all the migrants they covered. Mapping and risk assessment of migrants are needed to focus interventions on those migrants with high-risk behaviors. With individual risk assessment, many of the migrants currently reached might be found not to be high-risk for STIs and HIV transmission. Thus some migrants might not need STI services and ICTC, and the coverage for service uptake might change. However, focusing only on certain migrants in any workplace may stigmatize them and reduce uptake of services. Further, because of the NACO policy for focusing only on male high-risk migrants, the needs of migrant and other women laborers, who are frequently exposed to gender based violence and have wider reproductive health and family planning needs are not being met. This is a missed opportunity. Similarly migrant laborers may be faced with other, more immediate health problems such as industrial injuries. Under NACO protocols, addressing wider health and safety issues cannot be the entry point for migrant interventions although the health benefits might be greater and interest in health-seeking behaviors and adoption of lower risk sexual practices might be enhanced with such an approach. Barriers to migrants accessing services Migrants are some of the poorest and most oppressed people and are often unable to access services because they work long hours (8am to 8pm) in very demanding physical work. They are often unable to access services during their work day and are too tired in the evenings. Migrants want to take their day off to go to market and socialize, not going in search of STI services and ICTC. NGOs have adopted several approaches to overcome these barriers. They work through canteen personnel, petty traders near migrant camps and security guards at construction sites to make information accessible during break times (tea/snacks or lunch) and after work. As the time available during the breaks is short and migrants’ attention not focused, NGO staff have also successfully negotiated with some labor contractors and industry owners, to periodically release migrants during working hours to access health education and mobile van or health camp services in small groups. AVERT End of Project Evaluation 21 Migrants—not organized and not a stable population A further challenge to migrant interventions is that migrants have a tendency to shift from one construction site to another within the same area. This hampers follow up and support for high-risk migrants to continue to access services. Industrial migrant laborers may remain in an area for a long time but may visit their birthplaces in other states for festivals, such as Holi, and for the harvest. Additionally, as many migrants are not legally registered, they are not organized into labor unions. They live hand to mouth and have few rights. They are unable to access government services and social provisions. The Avert migrant interventions visited by the evaluators seem to be flexible in their approach and have been making effort to get to know the barriers to accessing services by the migrants they serve, and to tailor their approach to local needs. There are other migrants who come to the priority districts seasonally for agricultural work such as cutting sugarcane. However, this seasonal work is for less than 6 months duration and, under NACO policies, excludes these workers from migrant interventions although they may have high-risk behaviors. D. Is the Link Workers Program Making Progress Towards Achieving its Intended Goals, Objectives and Outcomes? What Have Been the Challenges? The Link Worker Scheme (LWS) is a 3-year program funded through NACO by the Global Fund to fight AIDS, Tuberculosis and Malaria. The purpose of the LWS is to provide a community centered model for prevention of HIV/AIDS in rural areas that are not reached by the TI programs. Specific objectives are to: • reach high-risk and vulnerable men and women in rural areas with information, knowledge and skills on HIV an STI prevention and risk reduction • increase availability and use of condoms among the high-risk and vulnerable populations • establish referral and follow up linkages to ICTC, STI and other services • create an enabling environment for PLHIV and their families in rural areas to reduce stigma and discrimination, through existing community structures Avert has implemented the LWS since April 2009, taking on implementation of the LWS at MSACS request after the initial lead agency failed to perform adequately. USAID is the only funder of its management support. Avert has scaled up the program into 24 districts in Maharashtra, in three phases to date. Currently, Avert’s memorandum of understanding (MOU) with MSACS is to implement the LWS from 1st December 2010 to 30th June 2011 and Avert has lead agency MOUs with district implementing NGOs for the same period. The LWS establishes village “knowledge information centers” branded as “Saiyukta” [United] at a site provided by the local community. These provide information on a general health issues including HIV and STIs, to avoid stigmatizing HIV and STIs. The LWS employs 1 link worker per three villages who • holds advocacy sessions at village level with the PRI, ASHA, AWW, ANM and sensitize them about the LWS • advocates with Chief Executive Officer (CEO) and DPM, NRHM about the LWS and seek to obtain letters from CEO Office to local PRI to encourage their participation • establishes Red Ribbon Clubs in each village, linked to Nehru Yuva Kendra AVERT End of Project Evaluation 22 • Co-ordinates with ASHA, ANM, AWW who have clear defined roles & responsibilities • Involves the Village Health Committee, PRI members • Undertake mapping of high-risk and vulnerable people in their villages and offer IEC to the high-risk and vulnerable people • organize health camps and coordinate with Mobile ICTC/STI services at villages to increase service uptake • link PLHA to locally available services and social schemes The LWS has reached 94% of the mapped high-risk population but this includes many villages where they are recorded as having reached 100% or more of the mapped risk populations. This calls into question the accuracy of the mapping and monitoring of some of the LWS activities. Overall, in Maharashtra, the program has registered 14,452 rural high-risk persons above those already known to TI programs and a further 496,541 vulnerable population and identified a further 6,973 PLHIV. Many of those identified by the LWS are rural FSWs. There are several challenges faced by the LWS. Salaries are low and do not attract males to the program. As a result, 70 - 85% of link workers are young females. It is unlikely that young women link workers are able to accurately and fully map MARPs such as MSM and they are not best placed to provide IEC/BCC to FSWs and MSM who are more effectively reached through PEs. A further challenge is that most of the peer services for FSWs and MSM are in urban areas and the DIC and support group for positive people are in the district centers. Thus referral and follow up is difficult for those identified in need by the LWS. The evaluators understand that some link workers are ASHA which might have both advantages and disadvantages. Training ASHA as link workers might make the interventions more sustainable after the end of the 3-year scheme. However, the link workers have enough responsibilities providing LWS services in three villages without also becoming an ASHA in a village. Overall, the LWS is spread too thinly and reaches out to relatively small numbers of high -risk and vulnerable people in rural areas through young women who are not best placed to influence risk and health-seeking behaviors in high-risk populations. However, the LWS is not collecting any data on the quality of the BCC provided and we cannot be certain that camps and mobile services are not testing many low risk persons again and again to reach their targets. 3) CAPACITY & SCALE-UP OF TARGETED INTERVENTION PROGRAMS A. Assess the Extent to Which NGO Capacity to Carry Out Core Skills in Targeted Interventions Has Been Fully Developed Avert capacity building has evolved over the life of the project. For many years, Avert commissioned training from a pool of Avert-approved trainers for NGOs providing TI services. This changed with the opening of the State Training and Resource Centre in 2010. Similarly the function of the TSU Capacity Building Officer has changed from designing training and being a trainer for TI NGOs across Maharashtra SACS supported AVERT End of Project Evaluation 23 districts, to undertaking training needs analysis and monitoring the quality of training provided by the STRC training institutions Capacity building for TI NGOs includes strengthening (1) financial management and reporting systems and (2) management information systems (MIS) and reporting, as well as (3) technical training related to prevention of HIV transmission in high-risk and bridge populations and referrals, communication skills, community mobilization and training of trainers (TO) for program staff and outreach workers. Some NGO staff have received training in proposal writing and grant applications. Some do not need that training as they have been successfully applying for grants for decades while others would like more. Monthly visits from the Avert DM and quarterly monitoring visits from Avert Society program and technical staff reinforce initial training with on the job coaching, refresher training and supportive supervision. These seem to be strongest for the financial management and MIS/reporting. NGO staff have been trained in TOT in an endeavor to cope with the large amount of training needed to scale-up community-based TIs. Some NGOs have stepped down training from trained PEs to newly recruited PEs. The challenge in terms of building institutional capacity to manage TI programs is the low salary and poor staffing levels imposed by NACO, particularly in the last year when staffing levels have been cut. Overworked staff who receive low salaries are constantly looking elsewhere for better employment opportunities. As a result, there is a high turnover rate of outreach workers. This situation is aggravated for the smaller NGOs that do not have the resources to retain and pay ORWs in the period—often three months—from the end of one financial year until their new sub-grant is signed and further funding flows from Avert. Thus, even with training of large numbers of NGO staff, including 760 ORWs, there is a constant need to train replacements as well as to train additional staff for scaling-up TIs. The gap between personnel trained and personnel needing training increases at the volunteer level because of the requirement to scale-up, most noticeably with TIs for migrants, where the VPLs may not be a stable population. At the end of the March 2011, there were 388 active PEs working with FSW, MSM and IDU communities of whom 303 have been trained. There were also 1710 voluntary peer leaders (VPLs) working with migrant populations, of which only 302 have been trained. It should be noted that as well as training NGO staff, Avert has been collaborating with MSACS and MDACS in the training of District government officers and leaders who are important stakeholders in creating an enabling environment for the TI programs. However, there is no systematic process in place for measuring the quality of TI services provided – particularly the BCC, as note earlier. If NGOS are reporting that they are making their targets, they are assumed to be providing good services to their communities. This is an important weakness in Avert’s TI core skills capacity building. B. How Effective Was the TSU in Supporting SACS to Scale-Up and Improve the Quality of the TI Program The Avert Technical Support Units (TSUs) were established in April 2008 to aid the implementation plan of Maharashtra and Goa State HIV and AIDS programs. The core areas of activity for the TSUs are providing capacity to MSACS/MDACS/Goa SACS for • Program support and monitoring of SACS TI programs • Capacity building for TI NGOs in SACS districts. Initially the TSUs undertook training through a panel of approved trainers, but since the STRC started in 2010, AVERT End of Project Evaluation 24 the TSU Team Leader Capacity Building undertakes training needs assessments for the STRC, commissions training from the STRC and monitors the quality of the STRC training when the trainees return to their jobs • On the job coaching of NGO field staff in completion of NACO monitoring forms, how to undertake “hot spot” analysis and how to do microplanning of their work • Evaluation of TI NGOs using the NACO self-reported formats every quarter. The additional capacity has allowed the SACS to scale-up TI interventions from 44 (2007-08) to 96 (2010-11) in Maharashtra and from 10 to 19 in Goa over the same period – particularly migrants and truckers interventions. It has also fostered more accurate data reporting with focus on actual target populations; the enhanced accuracy led to a reduction in the TI coverage in 2009-10. The TSUs also facilitated smooth transition of TI programs from other partners as their support to the SACS came to an end and planned transition of their programs to the SACS occurred. Challenges for the TSUs include their staff levels. Goa TSU has 2 POs for 19 TI programs and Maharashtra TSU has 10 POs for 103 TI programs in 33 districts [This includes Mumbai, and 2 of the 10 POs and 7 TI programs are specifically for Mumbai] There are challenges in (1) reaching hard to reach populations such as MSM (2) sex work is changing and now organized by cell phone rather than soliciting on the streets; also was predominantly brothel based but is increasingly becoming home based – not necessarily the FSWs home but in a house or hotel room rented for the purpose (3) identifying the high-risk persons among migrants – at first those with high-risk behaviors were not being reached and so the migrant TI has been changed. Some NGOs that were not reaching high-risk persons were closed out and others that were more successful were scaled up (4) IDU TIs are poor. There are 3 TI programs in Mumbai 1 in Maharashtra but mapping IDU is a challenge The TSU Team Leaders reported that they do all the QA/QI through the NACO developed NGO assessment sheets and the Program Officers [PO] field visit reports. They give grades to the TI NGOs based on a weighted score given to selected indicators. This provides the only measure of quality of the TI programs. Issues identified are followed up by the POs on their next field visit. In summary, the TSUs have contributed to significant scale-up of TIs, with quarterly technical review meetings with APD Avert, in Maharashtra and Goa. 4. Assessing the Effectiveness of Avert’s Various Management Systems A. Is There a System in Place to Assist the Staff in Capturing, Managing and Analyzing Program Data? Avert has an extensive system in place that assists staff in capturing, managing and analyzing program data. Much of this system is dual in nature, taking into consideration both NACO’s and PEPFAR’s protocols, formats and guidelines for collecting data and maintaining records. At the NGO level, PEs and ORWs collect basic data in their tracking sheets and weekly diaries. This information is then fed monthly to the NGO Project Coordinator who compiles it into the Monthly Technical Report. The District AVERT End of Project Evaluation 25 Manager both assists in data compilation, if required, and reviews this report before it is passed along to Avert in Mumbai. Data collected and maintained is heavily quantitative in nature and focused primarily on outputs: numbers of condoms distributed, contacts made, service referrals made, advocacy meetings held, etc. From field discussions, it is clear that this data collection and management consumes a significant portion of NGO staff time and is the major focus of monthly and quarterly visits from Avert Mumbai staff. The NGOs are clearly focused on the importance of M&E, or MIS as many know it, and most questions posed in the field were answered in the context of data collection. When asked about training received and ways in which it was found to be useful, most of the NGOs responded with details of their MIS training and how it has helped them to improve and carry out their data collection and reporting responsibilities. Most understood the importance of MIS and appreciated the Avert MIS system and the extensive assistance received from the District Manager and the Avert MIS staff during their monthly and quarterly monitoring visits. When asked about achievements and performance trends over multiple years, most were only able to pull out sheets of monthly figures. A few NGOs had sheets that tracked numbers and service trends by year, but it was not clear if these had been prepared by their own staff or forwarded from the Avert Society. Most NGOs said that Avert carried out the analysis of data and trends for them; several said they would like to be able to do it themselves. A few NGOs said they can look at their data, identify the gaps and figure out where the programmatic problems lie, but most said that Avert does this for them. B. Is There a Systematic Process for Ensuring Data Quality Control at all Levels of Implementation, Including Spot Checks? There is a systematic process in place for data quality assurance (DQA) and it is being used to ensure consistent and regular data quality both at the NGO and Avert Society level. PEs and ORWs are trained, re-trained and monitored in correct data recording and NGO Project Coordinators and MIS staff are equally well-trained and coached to do the first line of data quality control before the monthly data goes to the Avert Society in Mumbai. District Managers play a key role in assessing the completeness and quality of data compilation on a monthly basis. Avert’s MIS staff carry-out periodic spot checks of data in the field including annual data quality reviews or evaluations. Clearly a tremendous amount of training and supportive supervision goes into maintaining DQA under Avert. C. How Effective is the Avert M&E System in Tracking Progress? The Avert Society’s M&E system is effective at tracking progress against a large number of outputs and process indicators as required by both NACO and PEPFAR. It is much less effective at monitoring and tracking qualitative aspects of the program. Avert prepares quarterly NGO Assessments, using the NACO format, based on quantitative data submitted monthly by each NGO, whereby each NGO is given a composite score based on 13 self-scored indicators. While several of the indicators are reasonable measures of quality service delivery, e.g. number of MARPs undergoing HIV testing twice a year, the information gets buried into the composite score. About half of the NGOs when asked did not know their current score in terms of percentage or grade. Those who knew their scores could usually tell us how changes in numerical targets had decreased or increased their score over time, but none could cite individual improvement in indicators that had been measured in the composite scoring. AVERT End of Project Evaluation 26 Despite the fact that the 2005 Evaluation of the Avert Project noted that “a few key indicators should be developed at the NGO level to assess the quality of services delivered”, this was never done. The USAID-approved July 2006 Program Description for the second phase of Avert makes reference to a Performance Monitoring Plan (PMP) and indicators for reporting under the Missions strategic framework, but no PMP or results framework for the Avert Project was ever developed. This would have been an appropriate mechanism for tracking indicators such as number of MARPs accessing twice yearly HIV testing and other more qualitative indicators. While this particular data set was eventually dug out of the Avert MIS database at the request of the evaluation team, it was time and effort consuming and the data was unconsolidated across programs. Had key indicators such as this been incorporated into a functioning PMP, it would have made it much easier to track quality of service indicators as well as key performance indicators that were not tracked consistently over the ten years due to shifting priorities and guidance from both NACO and PEPFAR. As an example, USAID and the Avert Project missed the opportunity to monitor and track the quality of the BCC being given by the PEs and ORWs, and focused only on the quantity of BCC encounters. Above and beyond the M&E system, Avert has undertaken a number of large scale surveys such as the very valuable BSSs, operations research and secondary data analysis to supplement the quantitative measurement of the MIS, but the MIS system should also have focused more on qualitative indicators. The “qualitative narrative reporting” that NGOs are encouraged to do and which was reviewed in the field, was generally little more than a narrative summary of all the figures that went into their monthly data reports. Occasionally a success story would get written up by an NGO or by Avert for publication as a PEPFAR Success Story, but to date there has still been insufficient documentation of the qualitative accomplishments under the Avert Project D. What Were the Strengths and Weaknesses of The Avert Project Related to Governance Structures? Governance structures4 The Avert Society, which implements the Avert Project, is accountable to a seven￾member Governing Board [GB]. The GB is chaired by the Secretary for Health, GoM5 . A USAID representative was vice chairperson and the other members include a representative of NACO, the Project Directors of MSACS and MDACS, the Director Health Services, Government of Maharashtra, and the Project Director of Avert Society. In 2009, USAID stepped back to observer status on the GB. The GB should meet once a quarter, with decisions taken by consensus. The National Steering Committee, mandated in the original project agreement,6 is not referenced in 2006 Ninth Amendatory Agreement7 and presumably is no longer a governance structure. A Management Committee, consisting of Project Directors of MSACS and the Avert Society, and a representative of USAID, established by the GB in 2008 to approve new grants above Rs300,000 and contracts above Rs1,000,000 ceased functioning in 2009 and is no longer a part of the governance processes. GB approval is required for very many Avert management decisions: i.structure and staffing levels; ii. staff emoluments and terms and conditions for service; iii. appointment and annual renewal of contracts of key staff; iv. strategic plans, annual plans and budgets, quarterly statements of expenditures and six-monthly cash forecasts; v. overall criteria for award of sub-grants and contracts; AVERT End of Project Evaluation 27 vi. award of grants; vii. the progress M&E plan and schedule for all program components and subcomponents; and viii. the terms of reference of midterm and end of term evaluation of sub-grants and contracts, and of the overall program. Strengths The GB is well placed for monitoring the Avert Project performance, and for sharing experience and lessons learned with the other AIDS societies in Maharashtra, and potentially with other state societies through NACO. Guidance from the GB should ensure that the Avert Project implementation is in line with the national priorities and plans—currently the NACP-III—PEPFAR requirements, and USAID/India’s Strategic Objective 14, as well as complementing MSACS and MDACS programs. The Avert Project should be enabled to collaborate and coordinate its programs with MSACS and MDACS to avoid duplication of activities within Maharashtra, while capitalizing on its comparative advantages of flexibility, rapid response and technical capacity, including operations research capability, to fill important gaps and strengthen state efforts to reduce the transmission of HIV. A further strength is in encouraging local ownership and potential for sustainability of Avert Society achievements, and smooth transition of Avert Society programs once established and performing well to MSACS and MDACS. Weaknesses Requiring GB approval for management decisions at operational level—rather than, for example, having the Avert PD accountable to the GB for project performance, with GB approval required only for the annual plan and budget, and expenditure above an agreed ceiling—meant that project performance was greatly hampered by the GB not meeting to give needed approvals. The 2005 evaluation8 noted that the GB had not always functioned in the most efficient and supportive manner. This has continued throughout Phase II with postponement of GB meetings such that there has been only 1 meeting in the last year. Recommendations, for establishing a year in advance and adhering to a schedule of quarterly GB meetings, were not implemented. The 2009 management review clearly documents the deleterious effect of the dysfunctional governance on project performance.9 At the time of this evaluation, even though Avert is only 3 months from the end of the Phase 2 agreement (end of June 2011), Avert has not received instructions to transition its programs, including the TI programs, LWS, and STRC to MSACS although USAID has drawn up a detailed and cost-budgeted transition plan.10. This process should have begun at least a year ago to permit smooth transfer with a period of continued technical oversight after the transfer. The consequences Avert PD and staff have faced many frustrations related to obtaining the needed permissions to implement the project. The current GB chairperson has endeavored to enable the Avert Project by receiving documents and giving approvals as far as possible, for later GB ratification, to compensate for the lack of GB meetings. Avoidance of duplication of programming efforts happened largely as a result of a directive in 2009 when the current Director MSACS was appointed for “one district, one donor”. At that time there was an exchange of programs between partners and with MSACS so that each district has only one donor funding programs or it has MSACS programs.11 Although the Avert Phase 2 agreement ends June 30th, 2011 there is no plan in action for transitioning Avert programs and important functions such as the TSU and STRC to AVERT End of Project Evaluation 28 MSACS. USAID submitted the transition plan to NACO in June 2010, but a formal communication approving the plan was not sent to MSACS until the recent GB meeting held on April 28, 2011. Also in this GB meeting, USAID/NACO communicated to the board that a formal request has been sent to the GoI for extending the Avert Project until March 2012. Avert staff do not have the authority to implement a transition plan without prior GB approval of the plan which has not happened. Avert staff have heard informally from NACO that NACO has requested USAID to support an extension to the project. This has left a vacuum in project implementation/transition/closeout planning and some NGOs have refused to take continue sub-grants in the current Indian financial year (from April 2011) because of the uncertainty over funding. E. What Were The Strengths And Weaknesses Of The Avert Management Systems Including Project Planning And Review, Grants Management, Financial And Procurement Systems In Scaling Up The Project Activities? Avert Society Management Systems Although the 2009 management review states broadly12 that “There are several weaknesses in the management systems”, it only specifies the procedure for recruitment of senior positions—which is a GB issue rather than Avert project management systems issue. Several respondents and documents13 refer to “Avert robust project management systems”. Strengths of the Management Systems The Avert Project Phase II began with a Management Systems Manual, developed by a Management Agency that had been ready to be launched at the time of the evaluation in November 2005. Avert finance staff revised the financial procedures in 2005. Avert staff affirm that the manual has proven useful for ensuring continuity during staff turnover— although it should be noted that of the 29 staff named in the 2008-09 status report 14 are still with Avert. Over the last three years, Avert staffing has expanded but turnover doesn’t appear to have been notably high. Avert Society salaries are on USAID/India pay scales for skilled professionals, and are higher than salaries in the other societies. Avert asserts that it has, in line with its human resource procedures14, clearly written job descriptions detailing expectations of staff and thus obtains high performance rankings from its staff. The PD holds weekly meetings on Mondays for the team leaders and monthly meetings for all the staff. The APD holds quarterly technical review meetings with the TSU staff. The current PD and APD seem to have addressed the need for a cohesive Avert team, moving away from what was noted, in 2005, to be a structure of vertically-operating technical units. The “gulf between technical/program offices and financial management” noted by the 2005 evaluation15 is no longer apparent, although the Finance Manager position is currently open. Weaknesses of the Management Systems Need to Update Avert Management Systems and the Operational Manual There has been some evolution of management procedures and practices, but these have not been formally incorporated into the management systems or into the Avert Operational Manual. The Avert PD and staff do not have authority make the necessary incorporations. The NGO monitoring system in the operations manual doesn’t match what is happening in the field. The procurement rules in the manual, for example, state that there has to be an advertisement and a competitive tender for materials/services of value Rs1,000,000 or above—which was fine 10 years ago but needs to be increased to, AVERT End of Project Evaluation 29 say, Rs5,000,000 now. Requirements for obtaining 3 quotes for everything—including purchase of small items of stationery such as pencils—is overly cumbersome. Lack of Management Technology and Software Avert GB would not permit investment in project management technology such as video-conferencing or electronic data collection and transmission from the field to Mumbai. NGOs use paper-based systems with registers; data entry into Excel spreadsheets is thus an additional and frustrating task for NGO staff. Additionally, the GB would not approve purchase of appropriate software for analysis and use of data, even though India is technology superpower— a situation led by private sector companies in Maharashtra. Project Planning & Review Annual Action Plans16 Avert produces annual action plans that provide narrative on the context of the project and the different program components. The plans review achievements against targets for the previous year, set quantitative performance targets for the forthcoming year, and set out the budget for the year. However, there have been considerable delays in obtaining approval for the budget—which is beyond what the GB Chairman can approve, pending ratification by the GB. Avert staff report that the 2010-11 Action Plan was ready March 15, 2010 and presented to NACO. Although verbal assurances were received sooner, there were no funds released until June 18, 2010. Although Avert asserts that “the NGOs have been remarkable in coping with no funding” for the first three months of their financial year, several NGOs and a CBO complained to the evaluation team about the severe difficulties they face when there is no funding for staff salaries and volunteer allowances for three months. Avert Status Documents The 2005 evaluation referred to an Avert “Status Document, November 2005”17 and a similar document exists for 2008-0918 that includes a “Strategic Implementation Plan”. The latter presents the Avert Project with the language of objectives, intermediate results and outputs/outcomes although a results framework linking to the Mission SO14 is not included and the intermediate results are not defined. As the original project design did not have a hierarchy of results (a logical basis), and Avert activities have repeatedly changed direction in response to directives from NACO, it is not surprising that there isn’t a results framework. There was a missed opportunity to strategically redesign the project logically at the time of the agreement to implement Phase 2, which had a more focused and epidemiologically sound approach and was expected to make a significant contribution to the Mission’s HIV/AIDS IR under SO14. As noted in section 4.c) the targets presented in the Activity Planning Document for Avert Phase 219 were not reported to USAID for project monitoring purposes. Grants Management Avert staff attribute the success of their programs to the process for selection of NGOs (the Avert procedures are said to have been adopted by NACO for the NACP-III), to the participatory method for monitoring programs (participatory site visits), and to the Avert experience sharing and reviewing practices. NGO Selection Process and Scoring Avert uses a thorough process for the selection of NGOs for sub-granting. The process is outlined in Annex I. The fairness of the process is assured by rigorous, transparent AVERT End of Project Evaluation 30 implementation, involving both technical and procurement staff, to reduce the risk of cronyism or back handed payments for the award of grants. NGOs are scored annually using the NACO tool, and poorly performing NGOs have not been given grant renewals. Where there are no NGOS working in an area to provide services—for example with the MSM community—Avert has identified a CBO, established rapport, motivated staff to want to implement a DIC for their community and coached them through the grant application process. Many NGOs have had coaching or “hand holding” to help them conform to grant monitoring and reporting requirements. Program Supervision and Quarterly Review Meetings Targeted Intervention programs are reviewed monthly by the Avert DM. Previously they were reviewed and mentored by Program Officers in Mumbai. The appointment of the DMs in 2009 was frequently mentioned as a milestone by NGOs that noted that although they had always been able to call Mumbai, staff would be in meetings or out of the office, leading to delays in returning calls. DMs with cell phones are able to respond faster and are also able to advocate directly with the DAPCU when there are shortages of condoms, HIV testing kits or STI treatment kits. Additionally, DMs can borrow commodities from a program that has stocks to avert a stock run out in another program. The DM checks diaries and reporting as well as offering technical advice. There is a full system of reporting formats. The DM meets at least monthly with the DAPCU and several DAPCU District Program Officers. This indicated how important the relationship with the DM was for effective district coordination and timely reporting (to MSACS and NACO). Avert program and technical staff from Mumbai visit districts once a quarter to hold 2- day district review meetings which have both monitoring/reporting and technical content. A key feature of the visits is participatory site visits with the NGO staff to enable supportive supervision and mentoring to improve program interventions. NGO field staff mainly referred to receiving support with reporting when asked about supportive supervision by Avert. Although the LWS is a larger program with an annual budget Rs3,000,000 compared with Rs1,200,000 for the TI program, the level of supervision and support is lower with one Mumbai-based PO managing three districts. The LWS was added as an extra job responsibility for Avert staff who are funded through the USAID grant. Thus USAID is adding value to the Global Fund grant in Maharashtra. Annual Experience Sharing and Review Meetings Avert facilitates an annual experience sharing and review meeting for its grantee NGOs. Each NGO submits papers that are reviewed by an Avert technical committee, and the best are accepted for presentation or for poster display at the annual meeting. This is an important forum for learning lessons and sharing experience. Avert has followed up annual review meetings with success and failure studies to continue the learning process. As the current TI program and LWS are very new, there has not to date been time to identify good “success stories”. Financial and Procurement Systems Avert Society receives its funding through NACO on a revolving fund system. However, NACO procedures for replenishing the revolving fund are so lengthy that the Society’s bank account is often almost empty. AVERT End of Project Evaluation 31 The Operations Manual20 includes detailed sections on Accounting and Procurement, Sub-grantee Selection, and Sub-grant Management. These are a sound basis for financial management and procurement although there is some unnecessary bureaucracy in the need for 3 quotations for small items of stationery such as pencils. There is a need to update the manuals to match field practice and authority for this should be vested in the PD. Additionally, the cost of materials and services that requires competitive tendering needs to be raised to keep pace with rising prices. F. What Were the Strengths and Weaknesses of the Avert Leadership Team in Steering the Project? Leadership Gaps The 2005 evaluation documented leadership gaps during the first phase of the Avert Project21. 22 For much of the Avert Phase 2 implementation there has not been a Project Director. The current Project Director (PD) joined in July 2010. Nonetheless, the Associate Project Director stepped up to the plate with leadership that now continues as leadership from within the Avert team in support of the PD. Strengths Having the current Associate Project Director (APD) in post from March 2007—she joined the staff as a Program Officer (PO) towards the end of Phase 1—provided a degree of continuity to the leadership that had been absent during the first phase of the Project. That the APD had risen through the ranks from PO encouraged rapport with the program and technical staff; and, as her employment began during phase 1, she had experienced management turmoil and the disruption it caused. Further, the APD had a vision for what Avert Society could become and a determination to spur the staff into taking the Society forward. The current PD/APD leadership has demonstrated team building skills, and technical understanding of wide ranging fundamental issues for addressing HIV in Maharashtra. The evaluation team noted rapport and respect cementing project relations with MSM and transgendered communities. The leadership has worked to maintain cordial professional relations in the face of skepticism about the role of the Avert Society from MSACS. The technical strengths of the current Avert leadership was acknowledged by NACO23 in relation to the development of revised migrant guidelines. Weaknesses Over the life of the project, Avert Society leadership has not been able to strengthen gaps in areas such as strategic planning and developing a results framework; defining good minimum clinical care packages for FSWs and MSM regular health checks, establishing service delivery quality standards, programming for continued quality improvement (although data quality assurance is in place), and monitoring quality of service delivery beyond the NACO NGO assessment format. The repeated need for “fire fighting” in support of the NACP-III—hosting the TSU at the request of NACO, then in 2009 taking over the LWS24 after the failure of the first NGO lead agency, and in 2010 taking on the STRC function for MSACS—as well as repeated changes in programmatic direction from NACO, has diverted the leadership’s energies away from writing up the studies that Avert has undertaken and from undertaking other studies they have identified a need for. G.How Effective was the Coordination Between Various Partners Including MSACS, MDACS, Other Donors and Stakeholders in Maximizing Resources Through Complementary Planning and Avoidance of Duplication of Efforts? AVERT End of Project Evaluation 32 At state level As noted in 4.d), avoidance of duplication efforts happened largely as a result of a 2009 directive, when the current PD MSACS was appointed, for “one district, one donor”. At that time there was exchange of programs between partners and with MSACS so that each district has only one donor funding programs or it has MSACS programs. Although the geographic division is not so clear cut in Mumbai, donors there are supporting different interventions. When Avert took on responsibility for the LWS, it undertook an exchange of some program areas between its TI and LWS programs, in the few areas that had TI programs extending into rural areas. This action also reduced duplication of effort. At district level The establishment of DAPCUs may increase the opportunity for coordination and complementary planning. In practice, the DAPCUs appear to be focusing mainly on reporting, although one DAPCU reported that the district program officer (DPO) collects the details of every ICTC client who tests positive. The DPO then allocates follow up of the individual clients to the appropriate NGO program and he demands follow up reports on the positive individuals. While this would indicate that persons who test positive lose rights to privacy, it is a process that considers it important to follow up positive persons and link them to services provided by networks of PLHIV, DICs and CCCs. Several DAPCUs noted that the role of NGOs is essential for reaching high-risk and vulnerable persons as well as providing services for PLHIV because the DAPCU staff cannot fill that role. Avert’s deployment of DMs may support complementary planning to maximize synergies between Avert TI programs. The DMs are reported to assist with reducing stockout situations, borrowing test kits and STI treatment kits from other programs and having stocks to lend to programs in danger of running out of stock. That the Avert NGOs in a district know each other, allows for informal coordination that maximizes use of resources. For example, the evaluation team met an ORW from a migrant program who reported that she had brought migrants with symptoms of an STI into a DIC established by another organization to serve NBB FSWs. The best example of planning to maximize use of resources is in relation to the mobile ICTC/STI vans. The vans serve all the TI programs in a district—each district has one van and that van is hosted by one organization. The coordination of the vans schedules and the agreements on how often the van visits each TI program may differ by host organization. However, the systems seem to be working well. Unfortunately, there is a suspicion that there might be some duplication of effort between the Avert support to the network of positive persons in each district—for the provision of support groups, DICs and home-based care coordinated through the DIC—with the services provided by the CCCs supported by the KHPT. H. Were the Planning, Management and Coordination Systems Adequate to Ensure Coordination and Synergies of All the HIV Prevention Efforts? What Were the Challenges Faced in Coordination? Prevention HIV prevention is undertaken by PEs and VPLs working with the TI programs, and the link workers. “Prevention for positives” is a component of positive living encouraged by AVERT End of Project Evaluation 33 the networks of PLHIV. The TI programs and the LWS in a district essentially cover different populations with TI programs in urban areas and the LWS in more rural areas. There are four main strategies for prevention in Maharashtra i) consistent correct use of condoms in high-risk sexual encounters ii) reduction in STIs through regular medical checkups for FSWs and MSM iii) regular HIV testing for high-risk populations (every 3 or 6 months) and vulnerable populations including bridge populations (annually) iv) prevention for positives Some of the NGOs and CBOs are developing self help groups that have IGAs and this was specified by one MSM organization to be important for stopping young men from going into sex work. The emphasis on reducing dependence on sex work was not so obvious with NGOs working with FSWs. Planning, Management and Coordination District level NGO planning and coordination for prevention occurred as part of the monthly meetings with DAPCUs and with the DM. Additional activity occurred each year in the run up to World AIDS Day for high profile district activities that often spread through the whole of the first week in December. Avert staff reported they played a key role in coordinating the visit of the Red Ribbon Express but this was not recognized by NACO. All the Avert NGOs work with different populations but, as noted in 4. g) the evaluation heard of one example of referral of a migrant with symptoms of an STI to a DIC managed by a TI for NBB FSWs. An important factor in coordination of prevention was that Avert was managing the TI program as well as the LWS, and was providing training for the Link Workers, ORWs, PEs and VPLs—initially commissioning training and from 2010 through the STRC function that Avert took on. This ensures that the prevention strategies and messages are consistent throughout Avert districts. Challenges and Avert Responses A major challenge is the turnover of NGO staff. Outreach workers move on to better paid opportunities and the populations themselves are instable. Avert tries to use certificates; training opportunities and opportunities for paid sessions as resource persons for trainings for motivating NGO Project staff. Migrants move around from one construction site to another and from Maharashtra back to their home states for festivals (such as the recent Holi festival) and for harvesting. FSWs are also mobile and often move from brothel to brothel when police start harassing them. Sex work and MSM cruising has very much changed over the life of the project. There has been a reduction in street based soliciting and cruising and an increase in the use of cell phones and internet sites. Avert’s experience sharing meetings are important for managing this challenge. As NGOs develop effective new strategies for reaching sex workers and MSM, they need to share their experience with others working in the field to maximize effective prevention among these hard-to-reach high-risk populations. A major challenge to the prevention strategy was that NACO did not provide STI treatment kits until October 2010. Avert encouraged NGOs to “develop the habit of paying for treatment” by telling those who were diagnosed with STIs at the regular health check-ups to purchase treatment on the open market. A further challenge has been the shortage of public sector doctors in more rural areas. Avert has managed this challenge AVERT End of Project Evaluation 34 by arranging training in STI diagnosis and management for private practitioners near migrant camps. V. DISCUSSION OF THE THREE OVERARCHING EVALUATION QUESTIONS & CONCLUSIONS 1. HOW EFFECTIVE WAS THE AVERT SOCIETY MODEL, PARTICULARLY THE GOVERNANCE STRUCTURE, GOVERNANCE PROCESSES AND IMPLEMENTATION ARRANGEMENTS INCLUDING MANAGEMENT SYSTEMS, IN IMPLEMENTING THE AVERT PROJECT? As noted in the findings, section 4, Avert Society has achieved much and contributed greatly to implementing the NACP-III in its districts, as well as supporting MSACS programs in other districts through the TSUs, LWS and STRC. Project implementation has been enabled by Avert’s robust internal management systems including project annual planning, sub-grantee selection and sub-grant management, although as stated in Findings section 4.c, there are gaps in the development and implementation of a QA/QI system. However, project implementation has been hampered by inadequate funding flows from NACO through the revolving fund. As a result, Avert’s operating account has been almost empty during lengthy waits for NACO’s procedures for replenishment. Avert’s operations have also been hindered by unnecessarily burdensome procurement systems and low ceilings that have not been reviewed and revised since Avert’s operations manual was first developed. Moreover, implementation has been greatly hindered by the need to obtain external approvals for all sub-grantees, and awards of grants and contracts. The low level of authority vested in the Project Director obstructs rather than enables operations. Vesting authority for approval for sub-grantees, grants and contracts with a governing board external to Avert Society, in addition to the requirement for governing board approval of annual plans and budgets, has resulted in extreme delays that have severely compromised project performance. These delays and hindrances to the project outweigh the benefits of good internal management systems and the effective procedures to reduce corruption in selection of sub-grantees that Avert Society has put in place. The Avert governance structure, involving USAID, NACO, MSACS, MDACS and the Maharashtra State Ministry of Health, has delayed crucial decision-making through personal politics and power tussles unrelated to Avert and project implementation needs. USAID has taken proactive steps to improve governance. USAID conducted the Avert Project Management Review to address governance issues. In addition, USAID constantly advocated with NACO and the Government of Maharashtra to expedite decisions on fund release, conduct of GB meetings and on leadership issues. Notwithstanding, the Avert model has hampered, not enabled, implementation of the Avert Project. The project would likely have been more successful if it had not been AVERT End of Project Evaluation 35 held back by governance issues and changes in focus to remain in line with NACO policies and procedures. 2. WHAT ARE THE FUNDAMENTAL/STRUCTURAL ISSUES INCLUDING CHALLENGES THAT HAVE AFFECTED THE PROGRESS OF THE AVERT PROJECT? Avert has not been able to be an innovative project in developing and testing new approaches to fighting HIV in Maharashtra. This had been the intention expressed in the original intergovernmental agreement. Avert’s comprehensive district model for HIV prevention, care and support and treatment has had frequent changes imposed by NACO’s changing priorities and policies. Avert’s slum program was stopped even though Avert had undertaken valuable studies to identify which slum dwellers, including migrants, were at risk and most vulnerable. Avert’s work with prisoners was halted even though prisoners are known to practice high-risk behaviors (male prisoners having sex with men and use of injection drugs) and are subject to sexual violence that places them at greater risk of HIV transmission. Relatively recently, migrant TIs were started and more recently still the LWS was added. Thus the comprehensive district model has not been implemented long enough in any single form to permit full evaluation of the effectiveness of the model as a whole. Similarly, the heart was ripped from Avert’s integrated care approach for adults and children infected with, and affected by HIV, when its CCCs were transferred to KHPT. Avert has tried to manage the challenge of losing the CCCs by supporting a network of PLHIV in each of its districts to manage a DIC and support groups that link PLHIV to services including referrals of children affected by AIDS to NGO provisions in some districts. Whether this is efficient—there is suspicion that there is duplication of some Avert project efforts with KHPT provision through the CCCs—and whether it is the most effective model, cannot be assessed by this evaluation. There is no possibility for comparison between the current Avert model and the earlier model of having the networks of PLHIV, with DIC’s and support groups, linked to the CCCs. The fundamental, structural issue has been the institutionalization of Avert Society under NACO. NACO’s inflexible approach to uniform implementation of the NACPs throughout India—whereby its guidelines are used to cap interventions and services rather than provide the minimum standard—has meant that Avert project implementation has had to be in-line with and not deviating from the NACP-III and NACO’s policies and procedures. This has stifled all attempts at innovation and novel responses to HIV, the original concept for the Avert Project. In addition, it has wasted project resources and staff energies invested in approaches that were halted before they had been evaluated for effectiveness. 3. HAS THE AVERT PROJECT MADE A DIFFERENCE IN ITS PRIORITY DISTRICTS (WHERE IT IS THE ONLY DONOR-SUPPORTED PROJECT) IN INCREASING ACCESS TO AND UTILIZATION OF SERVICES; IMPROVING BEHAVIOR CHANGE; AND REDUCING HIV PREVALENCE? Two priority districts, Aurangabad and Nagpur, have received their only support over the past decade from the Avert Project. By comparing achievements in these two districts with the achievements in two comparison districts that share social and geographical characteristics, Parbhani with Aurangabad and Chandrapur with Nagpur, it is possible to AVERT End of Project Evaluation 36 get a sense of the comparative edge the Avert districts may or may not have gained during the life of the project. A comparison against various key indicators—HIV prevalence (ANC), trends in HIV testing and HIV positivity rates, as well as behavioral change as measured by CCCU—shows considerable similarity of performance between Aurangabad and Parbhani and between Nagpur and Chandrapur over the period of phase two of the Avert Project. Essentially there has been reasonable progress in all four districts. Clearly Avert has made a difference in terms of expanded coverage of MARP, mobilization, empowerment and education of high-risk communities for behavior change and increased use of services. Achievements under Avert have largely kept pace with those districts supported by other donors and/or MSACS. Forthcoming BSSs may well show an accelerated decrease in HIV prevalence in Avert focus districts. It is important to note that, in recent years, most districts in Maharashtra have benefited from Avert support to MSACS in implementing the LWS, hosting the TSU and overseeing the STRC functions. That achievements in Avert districts kept pace with other districts is a positive finding, but it is also a disappointing finding since there was an expectation that the innovations envisaged under the Avert Project would bring about a quick decline in HIV prevalence through greater utilization of prevention, care and support services. The advantage of these innovations should be evident now, after 10 years of project implementation. However, many of the innovations under Avert were simply never nurtured or even allowed to happen because they would have diverged from national guidelines. While national guidelines are essential to upholding standards, they should be a minimum standard and not used to cap innovations that might exceed those standards and bring about greater improvement. CONCLUSION This evaluation concludes that the Avert Society model has greatly limited the effectiveness of the Avert Project. Nonetheless, Avert staff have achieved project successes and made a difference in its priority districts of Nagpur and Aurangabad in the fight against HIV. They have done this without the performance management benefits of a PMP, and despite the lack of enablement from its governing board. In sum, crucial factors limiting the success of the Avert Project have been: 1. institutional – placing the Avert Society, and effectively the whole of the Avert Project, under NACO severely limited innovation and improvement of service delivery above the minimum standard defined by NACO policies and procedures. 2. governance – vesting critical authority in an external governing board whose members are not accountable for project performance, but whose inaction limited vital decision making, greatly delayed project implementation and project performance. 3. the lack of a framework for performance monitoring – a PMP – and regular review against the indicators and targets in a PMP resulted in no mechanism for project managers—within Avert Society and USAID—to ensure that implementation was in line with the agreed project objectives. 4. the omission of a QA/QI component to project implementation has resulted in crucial lack of data for measuring the effectiveness of project interventions and services and for monitoring improvements, for example in regards to behavior change communication—vital to reducing the risk behaviors and increasing the health-seeking behaviors that are required for reducing transmission of HIV. AVERT End of Project Evaluation 37 AVERT End of Project Evaluation 38 VI. LESSONS LEARNED GOVERNANCE AND FINANCING 1. Structuring a project with an external governing board (rather than a project technical advisory committee) with executive decision-making authority over project expenditure and operations opens project implementation to serious delays that compromise project performance. o Authority for appointment and line management of the Project Director should lie with one agency (with approval by USAID) o Annual plan and budget might be approved by donor & recipient government but senior project staff (PD, Operations Director/Financial Director) should have authority for operational financial approvals within the approved annual budget. 2. If project finances flow through a revolving fund, the fund has to be large enough to maintain operations throughout the period of auditing of expenditure and replenishment of funds. The fund amount should be reviewed regularly during the life of a project and increased as necessary. PROJECT MANAGEMENT AND OVERSIGHT 3. Projects need to have a logical structure with a hierarchy of expected results from which the PMP, with agreed performance targets, is constructed. Reporting against a PMP provides: i. more rigor to monitoring ii. a framework and data for evaluation iii. implementation focused on delivering agree results iv. a mechanism for project managers to monitor their own performance. o A project concerned with improving quality of services needs to be monitored against its achievement of improved quality services. PROJECT IMPLEMENTATION 4. Omission of a QA/QI component in i. NGO capacity building—including strengthening NGO management systems and staff training/supervision to include QA/QI, and in ii. monitoring NGO programs and services increases the potential for o misinformation and misunderstood BCC messages; and o perpetuation of hazardous clinical practices. 5. The Avert Project needed to develop quality indicators and track the quality of services its NGOs provided. 6. Avert society developed a well conceptualized model for District Comprehensive Services, but repeated changing of program components in response NACO directives wasted investment in program components that were stopped before they were evaluated for effectiveness. This was very dispiriting for program staff. o Evaluating the effectiveness of a complex model like the DCM requires implementation within an operations research framework, with collection of monitoring data against appropriate indictors for effectiveness from a baseline. Achieving measurable change as a result of community-based interventions is likely to take at least 3 years. o Most of the current Avert programs are only 2 years old and are not currently showing results at outcome or impact, although PEs and VPLs are clearly empowered by the experience. AVERT End of Project Evaluation 39 7. The TSU has contributed to significant scale-up of targeted interventions in MSACS districts throughout Maharashtra. There is a reasonably good system in place to track progress in these districts. 8. The mobile ICTC strategy has been highly successful in bringing STI/HIV services closer to hard-to-reach and vulnerable rural populations. The coordination of the mobile ICTC vans to support all the TI programs in a district is an excellent example of effective planning to maximize use of resources. Significantly the DAPCUs attribute increased uptake of HIV services to the mobile ICTC strategy. 9. The LWS, which is thinly spread, is targeting relatively small numbers of high-risk and vulnerable people in rural areas. It has weaknesses in its mapping, doesn’t evaluate the quality of the link workers BCC, and employs mainly young female link workers because the salaries are not high enough to attract males. This is unlikely to be an effective approach to prevention. Best practices for prevention among vulnerable and at-risk populations use peer to peer strategies. AVERT End of Project Evaluation 40 VII. RECOMMENDATIONS 1. It is crucial that a plan for transitioning Avert programs and important functions— such as the TSU and STRC—to MSACS is agreed upon and that the Avert Society receives the necessary authorizations to implement the transition plan over the remaining period of the project. 2. Future HIV/AIDS activities would be more effective if they are in the non￾governmental sector and outside the limitations and restrictions of the government system. The inflexible implementation of NACO policies, procedures and guidelines too often limited service standards rather than defined minimum service delivery requirements. 3. Project governance should not vest approvals for expenditure and implementation decisions within the annual plan and budget, in an external body whose members are not accountable for project performance. Decision-making within the annual plan and budget are better vested in the project director and an operations or finance director who are accountable for project performance. They might be guided by a non-executive, technical advisory committee. Annual plans and budgets should be approved by USAID/India with the Government of India for bilateral projects. 4. Performance monitoring plans, based on the hierarchy of results for a project, are important for monitoring performance against the agreed strategic objectives, indicators and targets. The hierarchy of results might be in the form of the results framework used for strategically planning the intervention, that contributes to USAID/India’s Mission results framework SO14; or, if used for designing the project, a logframe. 5. Future HIV/AIDS initiatives should ensure that QA/QI systems are addressed during the project design phase so that they contribute to and support the Project logframe, as well as the performance monitoring plans. Design of these systems should reflect a thorough review of QA/QI systems being utilized by other HIV/AIDS, BCC and broad￾based health projects in India as well as best practices world-wide. 6. Use of MIS for evidence-based decision making is more likely to be sustained after a project if NGOs are trained to do their own analysis of trends and programmatic challenges. During the remaining months of the Avert Project, value would be added if Avert trained and coached its sub-grantees in interpreting their MIS data and making evidence-based management decisions to improve their programs. 7. There is a real need in India to foster organizational cultures within the health and development sector that are concerned about the effectiveness of programs, services and interventions and improvement in quality. Technical assistance and training in development and monitoring of QA/QI should be included in any new HIV/AIDS assistance along with M&E indicators that measure quality of services and activities. 8. The quality of BCC with program beneficiaries should be a future focus area, since reduction in risk-taking and improvement in health-seeking behavior is directly linked to the knowledge levels of the beneficiaries. Consistent with recommendation 5 above, development, implementation and monitoring against QA/QI systems for BCC should be addressed during the project log-frame and PMP development during the project design process. AVERT End of Project Evaluation 41 9. NACP Phase Four should adopt the TSU, as this is an effective approach. It should also adopt the mobile ICTC strategy which Avert has demonstrated to be an effective use of resources to bring HIV/STI services to hard-to-reach vulnerable populations and increase the uptake of HIV services. 10. The Link Worker Scheme should be reviewed. Best practice for reaching MARPS is through peer to peer strategies. Many of the link workers in Maharashtra are reported to be young women and none seem to be peers of the target populations. AVERT End of Project Evaluation 42 1 NACO Press release 01 December 2010 (2010) http://www.nacoonline.org/upload/HomePage/NACO%20Press%20Release%20on%20HIV%20Estima tes.pdf accessed 05-08-2011 2 Drop Box presentation Maharashtra, source HIV Sentinel Surveillance 2008 data (unpublished) 3 http://www.nacoonline.org/About_NACO/ accessed 05-08-2011 4 Sources of information include: i. “Annex 1: Amplified Project Description.” In Project Grant Agreement between the President of India and The United States of America for Project “AVERT”. (1999) ii. Agreement Among The Government of the Republic of India, The Government of the United States of America and The AVERT Society. (2001) 5 currently represented by the Additional Chief Secretary, Government of Maharashtra, Public Health Department 6 Project Grant Agreement between the President of India and The United States of America for Project “AVERT”. (1999) 7 Ninth Amendatory Agreement to the Project Grant Agreement Between the President of India and the United States of America. (2006) 8 Evaluation of the AVERT Society Project Final Report. (2006) 9 Basavaraj, S., S. Gupta, J. Hayman, and S. Mohammed. Report on Management Review of Avert Project. (2009) 10 USAID presented the transition plan to NACO a year ago, but NACO has not accepted it. 11 As noted in 4g, there has been more recently some exchange of program areas by AVERT between its TI and LWS programs in the few areas that had TI programs extending into rural areas 12 Basavaraj, S., S. Gupta, J. Hayman, and S. Mohammed. Report on Management Review of Avert Project. (2009) 13 Including the “Scope of Work for Management Review of Avert Project”, and “COP FY10 AVERT IMBC Narratives” 14 Management Systems for AVERT Society. (Undated) 15 Evaluation of the AVERT Society Project Final Report. (2006) 16 The EOP Evaluation Team referred to: i. Annual Action Plan 2007-08; ii. Final Annual Action Plan 2008-09; iii. Annual Action Plan 2009-10 (Approved); iv. Draft Annual Action Plan 2010-10 17 The EOP Evaluation team has not had sight of that document 18 Status Document (2008-09) 19 “Activity Planning Document Avert Project – Phase II”. Attachment I to Action Memorandum: Approval of the Avert Project Phase II under Strategic Objective (SO) 14 “Improved Health and Reduced Fertility in Targeted Areas of India” 20 Management Systems for AVERT Society. (Undated) 21 Evaluation of the AVERT Society Project Final Report. (2006) 22 Annex I provides a table detailing the project leadership from inception to January 2009, taken from Basavaraj, S., S. Gupta, J. Hayman, and S. Mohammed. Report on Management Review of Avert Project. (2009) AVERT End of Project Evaluation 43 23 Evaluation team meeting with NACO, April 25, 2011 24 AVERT states that Maharashtra is the only state with one LWS managing partner; in consequence of the late start, Maharashtra was behind at the first LWS review but had caught up by the second review. AVERT End of Project Evaluation I AVERT END OF PROJECT EVALUATION ANNEXES AVERT End of Project Evaluation I APPENDICES A. DELIVERY ORDER STATEMENT OF WORK/FINAL SCOPE OF WORK AGREED WITH USAID B. PERSONS CONTACTED C. TOOLS D. EVALUATION CALENDAR E: CORRECT AND CONSISTENT CONDOM USE F: HIV PREVALENCE IN AVERT PRIORITY DISTRICTS G: IMPROVED PE/VCL PERFORMANCE FOLLOWING INTRODUCTION OF NEW NACO TRACKING FORMATS AND RETRAINING IN BCC H: COMPARISON OF PERFORMANCE IN AVERT DISTRICTS TO NON￾AVERT DISTRICTS I. AVERT SOCIETY’S PROJECT DIRECTORS J. REFERENCES Appendices order in the same sequence that they are mentioned in the text. AVERT End of Project Evaluation III USAID/INDIA Office of Population Health and Nutrition Evaluation of Avert Project ANNEX A: DELIVERY ORDER STATEMENT OF WORK I. Identification of the Task USAID/India seeks to evaluate the performance, impact of, and lessons learned from the HIV/AIDS Prevention, Care and Support activity implemented by the Avert Project in the state of Maharashtra. 2. Background HIV/AIDS Burden in India1 India has the third largest HIV/AIDS epidemic in the world. According to 2008 Government of India (GOI) national estimates, there are 2.27 million people living with HIV/AIDS (PLHIV) in the country, with 322,561 on antiretroviral treatment. However, because India has such a large population, the estimated adult HIV prevalence is only 0.29%. This low prevalence rate is misleading, given that even small increases in the HIV/AIDS rates in India could have global ramifications. While India’s national HIV prevalence appears to be declining, falling from an estimated 0.36% in 2006 to 0.29% in 2008, this national figure masks the more complex variation in state and district-level prevalence throughout the south and northeast of the country. India has a concentrated epidemic, with the highest prevalence among most-at-risk populations (MARPs): 4.9% of female sex workers (FSWs), 7.3% of men who have sex 1 Source: NACO HIV Surveillance Report 2008 AVERT End of Project Evaluation IV with men (MSM) and 9.2% of injecting drug users (IDUs) are estimated to be infected with HIV. The National AIDS Control Organization (NACO) has prioritized 195 of the country’s 611 districts based on prevalence rates. There is also a growing concern that interstate migration in India could be fueling the HIV epidemic, with districts that were low prevalence a few years ago now showing trends of higher prevalence, which experts feel needs further scrutiny. The GOI is now implementing the third phase of its $2.5 billion National AIDS Control Program (NACP-III), 2007-2012, a plan developed with input from the donor community, including critical support from the U.S. Government (USG). This GOI strategy outlines a decentralized response to the epidemic to deliver expanded prevention, treatment and care services, with the goal of integrating HIV/AIDS services within the National Rural Health Mission (NRHM) by 2012. The NACP-III has redoubled its efforts to expand services and tailor interventions to the unique epidemiological context of the epidemic in India. Given NACP-III’s current momentum and significant advances in scale up and capacity development at the central and state level, reversing the epidemic may prove to be within reach over the next five to ten years. III. Overview of USAID HIV/AIDS program USAID supports GOI efforts to reduce HIV/AIDS prevalence and mitigate the impact of the disease in the country. The overall Fiscal Year 2010 HIV/AIDS funding of USAID is $22.1 million; the program focuses on four priority states: Tamil Nadu, Maharashtra, Karnataka and Andhra Pradesh, with additional limited technical support to Kerala, Goa, Uttar Pradesh (UP) and Uttarakhand. USAID supports the bilateral AIDS Prevention and Control Project (APAC) in Tamil Nadu, the bilateral Avert Project in Maharashtra and the Samastha project in Karnataka and Andhra Pradesh, to implement comprehensive prevention and care and treatment programs. In addition, USAID supports the Samarth project, which provides technical assistance (TA) at the national level and in UP, as well as the Connect Project, which seeks to leverage and build public￾private partnerships to increase the use of prevention, care and treatment interventions. In its new strategy, USAID is moving toward strategic provision of TA and strengthening the quality of service delivery through strategic partnerships that leverage public and private resources. IV. The Avert Project The Avert Project is a bilateral agreement between the Government of United States and the Government of India. The project was signed in September 1999, with the aim of reducing the impact of HIV/AIDS in the state of Maharashtra. Overall, antenatal prevalence in the state is still above the national average (0.31%) for India at 0.54%, although there has been a decline among pregnant women attending antenatal clinics (ANC) from 1.5% in 2003. There are a number of factors that contribute to Maharashtra’s vulnerability to the HIV epidemic. It is bordered by other states that have well-established and growing HIV epidemics (Karnataka and Andhra Pradesh). There is extensive migration to and from these states, and there are major transportation routes connecting Maharashtra to them. Maharashtra is a major destination hub for migrants from various states of India. Additionally, Mumbai and several other districts have well￾recognized places where sex workers operate. AVERT End of Project Evaluation V The Avert Project supports the NACP and works in collaboration with the Maharashtra State AIDS Control Society (MSACS). MSACS is implementing a comprehensive HIV prevention, care and treatment program throughout the state of Maharashtra and is supported by the Mumbai District AIDS Control Society (MDACS). The Avert Project and the Bill and Melinda Gates Foundation (BMGF) are the two major programs that complement the efforts of MSACS in scaling-up HIV prevention, care and treatment programs. BMGF’s primary focus is supporting prevention programs among MARPs in 13 high-prevalence districts in Maharashtra. Additionally, UNICEF provides technical assistance on prevention of mother-to-child-transmission (PMTCT). The Clinton Foundation is supporting pediatric antiretroviral treatment (ART) services. The state also receives funds from the Global Fund to Fight AIDS, Tuberculosis, and Malaria for scaling-up integrated counseling and testing and care and treatment programs. In addition, USAID supports Johns Hopkins University (JHU) and the Hindustan Latex Family Planning Promotion Trust (this project ended in September 2010) to assist the state in designing behavior change communication and condom social marketing programs for MARPs. The first phase of the Avert Project was launched in November 2001 and ended on September 30, 2006. The project strategies included prevention (including workplace interventions), care and treatment, communication, research and capacity building. To implement the strategies, the Avert Project supported non-governmental organizations (NGOs), community based organizations (CBOs), People Living with HIV/AIDS (PLHIV) networks and institutions. In the first phase, Avert Project supported over 74 projects to implement targeted interventions (36) among MARPs, workplace intervention programs (11), care and treatment (23) and capacity building of providers (4). A mid-term external evaluation of the Avert Project was carried out in November 2005 to assess the progress of the project. The evaluation concluded that the Project has successfully established the HIV/AIDS program, developing close collaborative working relationship with MSACS, MDACS and other stakeholders in the state of Maharashtra. The evaluation also pointed out the need to strengthen the governance of the Avert Project, including regular governing board meetings to support scale-up and improved quality of HIV/AIDS programs. The second phase of the Avert Project was approved for the period October 2006 to June 2011. On March 7, 2008, NACO provided new policy guidance to the Avert Project focusing on saturating coverage of MARPs including migrants, implementing community mobilization activities to increase the uptake of counseling and testing, prevention of parent-to-child transmission of HIV (PPTCT), and care and support services in the five high-prevalence districts. The goals of the Avert Project in the second phase were to: a) scale-up prevention activities to support saturation-level coverage (85-90% ) of MARPs in five high￾prevalence districts, b) demonstrate models in community mobilization activities to increase the uptake of various care and treatment services in five districts, and c) scale-up workplace interventions in the entire state. NACO has also mandated the Avert Project to implement the Technical Support Unit (TSU) program in Maharashtra and Goa States. The purpose of the TSU is to assist MSACS, MDACS and the Goa SACS to scale-up evidence-based programs and improve the quality of targeted intervention (TI) activities. NACO also selected the Avert Project as the nodal agency to implement the Global Fund-supported link workers program in AVERT End of Project Evaluation VI the state of Maharashtra. In addition, the Avert Project was tasked by NACO to implement the State Training Resource Center (STRC) for training all the non￾governmental organizations (NGOs) in the state on core skills in TI programs for MARPs. Governance of the Avert Project The Project is guided by a Governing Board chaired by the Secretary (Public Health), Government of Maharashtra, with representation from NACO, MSACS, MDACS, the Director of Health Services, the Project Director of the Avert Society, a member living with HIV, and an NGO member. USAID is the observer on the board. The Board meets quarterly to guide policies and plans, including approving grants and procurements for various prevention, care and treatment activities, ensuring they are in line with and support national and state priorities. A National Steering Committee (NSC) has been established to provide high-level policy guidance and review the progress of the Avert Project. The NSC is chaired by the Director General of NACO and members include a representative of the Department of Economic Affairs (DEA), a representative of the Government of Maharashtra, the MSACS Project Director, MDACS Project Director and a USAID representative. The NSC is mandated to meet twice a year. Key Ongoing Activities:  Support and strengthen the capacity of over 50 NGOs and community-based organizations (CBOs) to saturate the coverage of MARPs including migrants, to promote behavior change, sexually transmitted infection (STI) treatment and counseling and testing.  Implement link worker (LWS) programs to mobilize MARPs in rural areas to utilize prevention, counseling and testing, PPTCT, care and support and ART services.  Support eight drop-in centers to implement home-based care and provide HIV/AIDS services to orphans and vulnerable children (OVC) infected and affected by HIV/AIDS.  Scale-up and strengthen workplace intervention programs with organized and unorganized industrial sectors vulnerable to HIV/AIDS.  Create models of HIV prevention programs among short-stay migrants and support replication of the model at the state and national level.  Conduct activities to ensure a strong evidence base for planning and implementing HIV programs, including mapping, behavioral surveillance surveys, studies on access and quality of services, and needs assessments to support demand generation.  At the TSU, build the capacity of MSACS, MDACS and the Goa SACS in grants management, monitoring, institutional capacity building, mainstreaming and public-private-partnership programs.  Support the STRC to train NGOs on core skills in prevention utilizing the nationally approved training modules. Evaluation Scope of Work A. Purpose USAID/India intends to carry out a final evaluation of the Avert Project to assess and document the successes, failures and lessons learned in complementing the efforts of the Government of Maharashtra to reduce HIV prevalence in the state of Maharashtra. AVERT End of Project Evaluation VII B. Statement of Work This statement of work (SOW) is for a comprehensive evaluation of the Avert Project, including the appropriateness of the project activities in achieving the objectives, the level of impact considering the contribution by the project, and future directions. Critical stakeholders will be involved during various stages of the review process as appropriate. The team will gather both qualitative and quantitative data based on the following evaluation questions. Overarching evaluation questions 1) How effective was the Avert Society model, particularly the governance structure, governance processes and implementation arrangements including management systems, in implementing the Avert Project? 2) What are the fundamental/structural issues including challenges that have affected the progress of the Avert Project? 3) Has the Avert Project made a difference in its priority districts (where it is the only donor-supported project) in increasing access to and utilization of services; improving behavior change; and reducing HIV prevalence? In answering these questions, the evaluation will: 1) assess the effectiveness of the Avert Project in terms of process, outcomes and impact: a) saturating coverage of MARPs through scaling-up targeted interventions for prevention, care, support, and treatment b) increasing consistent condom usage among MARPs c) improving quality of services d) reducing HIV prevalence and, e) Has the Avert Project made a difference in priority districts? f) How successful were the efforts of the Avert Project in developing community-based organizations and empowering MARPs to access services? (g) How effective was the Avert Projecet in developing a strong peer education program to deliver prevention and care services at community level? 2) Assess the success of the program interventions and models for prevention and care developed by the Avert Project: a) the District Comprehensive Prevention, Care Support and Treatment program in scaling-up access to quality prevention and care services by MARPs and their partners. What have been the challenges? b) the Integrated care model providing care and treatment services to adult and children infected and affected by HIV/AIDS. What has been the challenge for integrating adult and child care? c) HIV prevention for migrant populations. What have been the challenges? and, d) is the Link Workers Program making progress towards achieving its intended goals, objectives and outcomes? What have been the challenges? 3) a) Assess the extent NGO capacity to carry out core skills in targeted interventions has been fully developed. AVERT End of Project Evaluation VIII b) How effective was the TSU in supporting SACS to scale-up and improve the quality of the TI program. 4) Assess the effectiveness of the Avert Society various management systems—such as grants management, finance and monitoring and evaluation—in implementing prevention, care and treatment programs (a) How effective is the Avert monitoring and evaluation system in tacking progress? (b) Is there a system in place that assists the staff in capturing, managing and analyzing program data? (c) Is there a systematic process for ensuring data quality control at all levels of implementation, including spot checks? Does the process include both quantitative and qualitative methods? (d) What were the strengths and weaknesses of the Avert Project related to governance structure and processes? (e) What were the strengths and weaknesses of the Avert management systems including project planning and review, grants management, financial and procurement systems in scaling-up the project activities? (f) What were the strengths and weaknesses of the Avert leadership team in steering the project? (g) How effective was the co-ordination between various partners including MSACS, MDACS, other donors and stakeholders in maximizing resources through complementary planning and avoidance of duplication of efforts? (h) Were the planning, management and co-ordination systems adequate to ensure co-ordination and synergies of all the HIV prevention efforts? What were the challenges faced in co-ordination? 5) Document the programmatic challenges and lessons learned in implementing the Avert Project 6) Make suitable evidence-based recommendations for the future directions of USAID HIV/AIDS support in the state of Maharashtra. A. Duration: The duration of the evaluation will be for six weeks starting from the second week of March to the last week of April. D. Methodology The evaluators should utilize a range of possible methods and approaches for collecting and analyzing the information required to assess the evaluation objectives. Data collection methodologies will be discussed with, and approved by, the USAID/India HIV/AIDS team prior to the start of the assignment. Desk review of documents USAID/India will provide the team with all relevant country and project specific documents including proposals, evaluation reports and other relevant documents for conducting this desk review. The evaluation team is expected to collect and collate relevant international documents, reports, and data, and all team members are expected to review these documents in preparation for the team planning meeting. This desk AVERT End of Project Evaluation IX review will help to organize the materials for the external evaluation team analysis and review of progress to date, and facilitate their utilization during the field work, analysis and report writing stages. Data sources Data sources that the team will be expected to utilize, review and analyze include the project proposal, annual work plans, state annual action plan, sentinel surveillance reports, state data triangulation report, behavioral surveillance surveys, NGO evaluation reports, and other project-related documents and reports. Additional relevant documents related to HIV programming in India may be utilized as supporting documents. Team Planning Meeting (TPM) A two-day team planning meeting will be held by the team at an offsite location before the evaluation begins. This will be facilitated by the team leader, and will provide the Mission with an opportunity to present the purpose, expectations and agenda of the assignment. The evaluators shall come prepared with a draft set of tools and guidelines and a preliminary itinerary for the proposed evaluations. In addition, the TPM will also: • Clarify team members’ roles and responsibilities • Establish the timeline, share experiences and firm up the evaluation methodology • Finalize the methodology guidelines including tools and questionnaires to be used by the team. • Discuss and finalize evaluation questions based on the SOW Site Visits and Interviews • Conduct a thorough review of the Project through site visits and interviews. • Interviewees will include key members from all stakeholder groups, including NACO, the Government of Maharashtra, MSACS, MDACS, the Avert Project, other donors and partners in HIV/AIDS control, USAID and beneficiaries. • Interview questionnaire to be prepared in advance and finalized during the TPM. • Site visits will be planned taking into consideration factors like geographical diversity, representation of various beneficiary groups, and scale of interventions. • The team will evaluate state and district level periodic reports to verify results for the indicators. E. Composition, Technical Qualifications and Experience Requirements of the Evaluation Team USAID seeks a four-member assessment team (two international and two local members) comprised of a Team Leader/Senior Technical HIV/AIDS Expert/Health and HIV/AIDS Analyst, an Evaluation Methods Specialist, a Senior Public Health Specialist/Management Expert and a Public Health Specialist/Health and HIV/AIDS Analyst. All team members must have extensive HIV/AIDS program management, technical or implementation experience, familiarity with USAID’s objectives, approaches, and operations, and prior evaluation/assessment experience. The team will have experience in planning interventions for most-at-risk populations (MARPs) and needs to have collective expertise between them in programming for specific vulnerable populations such as female sex workers, MSM and IDU. All team members will need to have technical expertise including understanding of the HIV context in concentrated epidemics, with global knowledge on issues related to HIV prevention and/or care, AVERT End of Project Evaluation X support and treatment issues. Prior experience in India will be an asset; knowledge of the Indian National AIDS Program is desirable, though not essential. Collectively, the team must have experience in evaluating HIV/AIDS programs worldwide. In addition, individual team members should have the technical qualifications and required experience identified for the specific position below: 1. Team Leader/Senior Technical (HIV/AIDS) Expert/Health and HIV/AIDS Analyst (international): This Team Leader/Senior Technical (HIV/AIDS) Expert/Health and HIV/AIDS Analyst in the field of international HIV/AIDS prevention, care and treatment has an excellent understanding of global HIV/AIDS strategies and knowledge of the Indian epidemic and programs. Specifically, s/he should have an excellent understanding of the drivers of HIV infection in concentrated epidemics, with prior work experience in designing, monitoring and evaluating HIV/AIDS programs for specific most-at￾risk populations. Additionally, s/he should have proven experience in leading and managing large-scale evaluations of various HIV/AIDS programs throughout the world. S/he should have knowledge and experience on technical support strategies for strengthening the state’s capacity for an effective response to the HIV/AIDS epidemic. S/he should be familiar with the functioning of large donor funded programs in India. The person must have the ability to lead a diverse team of technical and management experts, and to interface with various stakeholders ranging from governmental to non-government organizations and donors, beneficiaries, etc. A minimum of 15 years of experience in the design, management and evaluation of HIV/AIDS prevention and control programs is required (LOE up to 34 days). 2. Evaluation Methods Specialist (international): This expert will have deep knowledge of evaluation methodologies and their practical applications. A minimum of seven years of experience in strategic planning, surveillance, operations research, and/or monitoring and evaluation of global and national HIV/AIDS programs is required. (LOE up to 30 days). 3. Senior Public Health Specialist/Management Expert (local): This Senior Public Health Specialist/Management Expert should be a management expert with extensive experience with USAID project design, implementation, and evaluation. The person should have an excellent understanding of USAID operational, management, and technical approaches. S/he should have thorough knowledge of project governance of large donor funded programs, including those that are managing a network of NGOs and institutions, working with government counterparts, as well as the various management issues related to such projects. In addition, the specialist should have knowledge and experience of HIV/AIDS prevention, care and treatment activities. A minimum of 12 years of experience in the design and management of HIV/AIDS prevention and control programs or health programs is required. Having knowledge and understanding of the Maharashtra State HIV/AIDS program and government systems would be an added advantage (LOE up to 30 days). 4. Public Health Specialist/Health and HIV/AIDS Analyst (local): The Public Health Specialist/ Health and HIV/AIDS Analyst should be an expert in HIV/AIDS prevention programs focused at concentrated epidemics. The specialist should have experience with the country specific HIV/AIDS prevention and control strategy and its approaches. Specifically, s/he should have AVERT End of Project Evaluation XI an excellent understanding of the drivers of HIV infection in concentrated epidemics, with prior work experience in designing, monitoring and evaluating HIV/AIDS programs for specific most-at-risk populations. A minimum of seven years of experience in the design and management of HIV/AIDS prevention and control programs is required. Having knowledge and understanding of the Maharashtra State HIV/AIDS program and government systems would be an added advantage (LOE up to 30 days). Summary Table: Labor Labor Category Level Maximum LOE Senior Technical (HIV/AIDS) Expert/Team Leader 1 34 Evaluation Methods Specialist 1 30 Senior Public Health Specialist 1 30 Public Health Specialist 1 30 In addition, each team member should have, at minimum, the following skills and experience: 1. An understanding of the country context. 2. An advanced degree in Public Health, Social Sciences, Business Administration, or other relevant course of study. 3. Demonstrated experience in designing, monitoring and evaluation of public health programs, especially HIV/AIDS programs that involve government engagement and partner management. 4. Very strong competencies in written and oral communication, as well as strong skills in relevant computer software packages (Word, Excel, PowerPoint, etc.). 5. Demonstrated knowledge of USAID policies and procedures. 6. Ability to work effectively in teams, and in communicating with a diverse set of professionals and stakeholders. The Senior Technical (HIV/AIDS) Expert will serve as Team Leader, and will be responsible for coordinating evaluation activities and ensuring the production and completion of a quality report, in conformance with this scope of work, which may become a public document for distribution among the program’s key stakeholders, including high-level U.S. government policy makers and officials, host country government officials, private sector and NGO leaders, and other audiences. In addition to proven ability to carry out this leadership role, produce a high-quality report, and present the findings effectively for this technically and logistically complex program, he/she should have substantial and demonstrated expertise in evaluation techniques involving projects which include technical assistance, training, advocacy, and partnership components. The Team Leader will be at a very senior level, with extensive experience in HIV/AIDS prevention, care and treatment programs, and must have excellent English language skills (both written and verbal), as s/he will have the overall responsibility for pulling together the different elements of the assessment into the final report. S/he will agree to fulfill the responsibilities in approximately six weeks, spending up to four weeks in-country, and will play a central role in guiding the evaluation process. The Team Leader may hold a AVERT End of Project Evaluation XII conference call with core team members and USAID/India representatives before and after the visit to India, if needed. Relationships and Responsibilities Overall Guidance: The USAID/India Health Evaluation Specialist, in conjunction with the Avert Project Activity Manager, other key HIV team members and the Regional Office of Acquisition and Assistance (ROAA), will provide overall direction to the assessment team. Responsibilities The Contractor will be responsible for: • Obtaining visas and country clearances for travel for consultants. • Coordinating and facilitating assessment-related field trips, interviews, and meetings in conjunction with the USAID, and Avert Project. • Submitting a budget for all estimated costs incurred in carrying out this review. The proposed cost may include, but is not limited to: (1) international and in-country travel; (2) lodging; (3) M&IE; (4) in-country transportation; and (5) other office supplies and logistical support services (i.e., laptop, communication costs, etc.) if needed. • In-country logistics to include transportation, accommodation, communication, and office support, etc. G. Reports and Deliverables The Team will provide separate sets of the deliverables mentioned below, for the evaluation of Avert Project. 1. Draft Work Plan and Pre-Departure Briefings. The evaluation team will develop a draft work plan prior to departure from Washington, D.C. The team will meet with USAID/India and other relevant contractor staff for at least three working days prior to departure for the field. 2. Mid-Point Review/Briefing: The evaluation team will provide a mid-point briefing to the USAID/India team, including evaluation and technical members, to clarify any outstanding queries that may have emerged since the initiation of the evaluation process. If this is not feasible based on scheduled field work, the Team Leader will submit weekly progress reports to the COTR via email by OOB Monday. 3. Oral Presentation. The evaluation team will provide an oral briefing on its findings and recommendations to relevant staff in the field, to GOI and state government officials, and to USAID staff at the conclusion of the visits to the various implementing partners. The evaluation team will be required to debrief the Mission Director and Deputy Mission Director on the observations and recommendations. 4. Reports: The evaluation will be required to submit the following reports: AVERT End of Project Evaluation XIII a) Draft Report. The evaluation team will present a draft report of its findings and recommendations to the USAID/India HIV Point of Contact (POC)/Activity Manager and COTR before the oral de-brief and return to the United States. b) Final Report. Submission of a final report in print (one copy) as well as an electronic version in Word X version shall be submitted within five working days following receipt of comments from USAID and its implementing partners. The final report will be provided to the COTR, USAID/India HIV POC and to PPC/CDIE/DI. The final report should include an executive summary of no more than three pages, a main report with conclusions and recommendations not to exceed 20 to 30 pages, a copy of this scope of work, evaluation questionnaires used to collect information on each of the program components, and lists of persons and organizations contacted. The final report, with executive summary and in electronic form, must be received by the COTR and USAID/India HIV POC within seven working days after receiving the final comments on the draft evaluation report from USAID/India team. AVERT End of Project Evaluation XV ANNEX B: PERSONS CONTACTED FOR THE EVALUATION INDIA New Delhi National AIDS Control Organization Ms. Aradhana Johri, IAS, Addl. Secretary U. S. Agency for International Development Ms Erin Soto, Mission Director Ms Elizabeth Warfield, Deputy Mission Director Mr Stephan Solat, Deputy Director, Office of Population Health and Nutrition Ms Elizabeth Callender, Program Officer Ms Amy Wielkoszewski, Project Development Officer Dr. V. Sampath Kumar, Project Management Specialist Dr Charushila Lal, Program Development Specialist AVAHAN – India AIDS Initiative, Bill & Melinda Gates Foundation Mr. Alkesh Wadhwani, Deputy Director Maharashtra Government of Maharashtra Mr. Jayant Kumar Banthia, IAS, Addl. Chief Secretary, Public Health Department Maharashtra State AIDS Control Society Mr. Ramesh Devkar, IAS, Project Director Goa State AIDS Control Society Dr. Padwal, Project Director Mumbai District AIDS Control Society Dr. Shantaram S. Kudalkar, Project Director Avert Society Ms. Smriti Acharya, Project Director Ms. Anna Joy, Associate Project Director Ms. Minati Sahoo, Capacity Building Specialist Ms. Anjana Palve, Care & Support Specialist Ms. Amita Abhichandani, TI Specialist Ms. Arupa Shukla, Communication Specialist Dr. Rajrattan Lokhande, Program Officer, MIS & SI Ms. Bhavika Badiani, District Manager, Thane Mr. Sandeep Palande, District Manager, Thane Mr. Siddharth Bhotmange, District Manager, Aurangabad & Jalna Mr. Yogesh Patil, District Manager, Solapur Mr. Nitin Bhowate, District Manager, Nagpur Mr. Santosh Suryavanshi, Program Officer, LWS AVERT End of Project Evaluation XVI Mr. Vijay Dhulla, Finance Officer Dr. Limaye, Consultant STI Program Mr. G. S. Shrinivas, Team Leader, Overall & Strategic planning, TSU Maharashtra Ms. Asha Vernekar, Team Leader TI, TSU Goa Aurangabad District District Health Officials Mr. Rahul Mahiwal, Chief Executive Officer Dr. D. N. Patil, Civil Surgeon Ms. Rohini Lahani, District Program Manager, NRHM Ms. Rui Siddique, DPO, DAPCU Dr. Shilpa Y Parwar, Sr. Medical Officer, ART Center Council for Rural Technology and Research Institute Mr. Chandrakant Ganvir, Project Director Mr. Kapil Ingle, Project Manager Gramin Vikas Sanstha Mr. Narhari Shivpure, Project Director Mr. Bharat More, Project Manager Marathwada Gramin Vikas Sanstha Mr. Appashaeb Ugale, District Resource Person, Program Ms. Rashmi Pund, District resource Person, Training Mr. Vithal Ramnath Bobade, RRC Chairman, Wahegaon village Mr. Dnyandeo Kanase, Link Worker, Wahegaon village Ms. Pramila Misar, ASHA worker, Wahegaon village Ms. Rohini Joshi, AWW, Wahegaon village Ms. Sunanda Khojandar, AWW, Wahegaon village Ms. Vasudha Chitte, ANM, Wahegaon village Prerna Samajik Sanstha Ms. Kadubai Bale, Project Director Mr. Sandeep Bhadange, Project Manager Udaan (Prabhat) Mr. P.V Ramesh, Project Manager Jalna District District Health Officials Ms. Bhagyashree More, DPO, DAPCU Dr. Manish Sahane, Sr. Medical Officer, ART centre Gramin Vikas Mandal Mr. Syed B., Project Director Mr. Yusuf Shaikh, Project Manager, Migrants Project Mr. Syed Hami, District Resource Person, Program, LWS Dr. Pradeep Gaikwad, Incharge, Mobile ICTC AVERT End of Project Evaluation XVII SETU Charitable Trust Mr. Sachin Panchal, Project Manager Thane District District Health Officials Dr. Sameer Bansode, DPO, DAPCU Mr. Ashok Deshmukh, District Supervisor Committed Communities Development Trust Ms. Tinku Biswal, IAS, Chief Executive Officer Ms. Chhaya Rade, Project Co-ordinator Community Aids and Sponsorship Program (CASP) Ms. Rekha Raje, Project Manager Family Planning Association of India Dr. Sheshgiri Rao, Project Director Mr. Vinayak Patki, Project Manager Ms. Pallavi Kale, Project Manager Humsaya trust Mr. Mangesh Manjarekar, Project Director Mr. Nishant Chotankar, Project Manager Network in Thane by People Living with HIV (NTP+) SHAPATH DIC Mr. Sachin Kulkarni, Project Co-ordinator Mr. Ashok, Founder Member, Peer Counselor Sankalp Rehabilitation Trust Mr. Appasaheb Mhaske, Project Manager Nagpur District DAPCU Mr Ganesh Parihar, District Program Officer Avert Society Mr Nitin Bhowate, District Manager Bhartiya Adim Jati Sevak Sangha(BAJSS) Ms Subhangi Kakre, Ex-Project Coordinator Mr Nirmal, Project Coordinator And ORWs & VPLs for migrant project Comprehensive Rural Tribal Development Programme (CRTDP) Mr I K David, Secretary CRTDP, Project Director Ms Yogita Ganvir, Project Coordinator And ORWs migrant program AVERT End of Project Evaluation XVIII Indian Institute of Youth Welfare (IIYW) Ms Vijaya Shah, Project Director Ms Varsha Pagale, Project Manager Dr Sandeep Bhende, Project Medical Officer (part-time) And ORWs for NBB Female Sex worked Project And ORW for Migrant Program in Kalamna Market Indian Red Cross Society (IRCS) Dr R P Singh, Project Director And ORWs and PEs of BB-FSW program Manav Vikas Bahudeshiya Mr Shekhar Giradkar, Project Manager Ms Sunjadip Kamve, ORW And PEs NBB FSW project Sarathi & MSM DIC Mr Anand Chandrani, Founder and Project Director And members of Sarathi Swami Vivekananda Medical Mission & construction site Dr Urmila Kshirsagar, Project Director Migrant Project Dr Ravindra Kshirsagar, Snr. Medical Officer SVMM Mr Ram Gopal Pandey, VPL migrant project Vaibhavi (Sanjeevan) NNP+, DIC, TAAL pharmacy Mrs Babita Soni, President Sanjeevan And members of NNP+, Beneficiaries of the DIC YMCA Link Worker Scheme Ms Madhuri, District Resource Person-Program And members of the community Solapur District DAPCU Dr Sayaji Gaikwad, District Program Officer Avert Society Mr Yogesh Patil, District Manager Seva Dham Trust and construction site, migrant project Mr Mallinath, Project Manager Mr Raju Manik, Counselor Ms Mahadevi, ORW Mr Sushil Bansuri, ORW Mr Prashand Pavand, ORW Mr Shah Alam, Volunteer Peer Leader Solapur Zilla Samajik Karya Samiti, power plant construction site AVERT End of Project Evaluation XIX Mr Girish Kolapure, Project Coordinator Mr C M Zalake, Counselor Mr E Krishna Rao, Volunteer Peer Leader Migrant Project Mr MS Kakra, Deputy Manager Power Project [employers of migrant labor] Satara District DAPCU Dr Kiran Jagtap, DPO (acting), Addiction Specialist Mr Pundalik Patil, District Supervisor ICTC ART Centre Dr Mrs Mandala Kanase, Senior Medical Officer Bel Air Hospital, STRC Fr. Benny, Project Coordinator – STRC Mr Jacob Joseph, Asst Administrator Kulkarni Charitable Trust – Work Place Initiative Dr. Mrs. Sheetal Kulkarni, Project Coordinator Dr Prakash Kulkarni, Project Director And representatives from local industries and Scooter dealership Network of Satara by PLHIV (NSP+) and Support Group Ms Karuna Pawar, President NSP+ Mr Vinayak Main, Project Coordinator, DIC Ms Deepali, Counselor, Employed at DIC Satara LWS Mr Sushil Mane, District Resource Person LWS AVERT End of Project Evaluation XXI ANNEX C: TOOLS DEVELOPED FOR THE EVALUATION 1. KEY INFORMANT INTERVIEW QUESTIONS FOR NACO 1. What has the Avert Project achieved? Probe: impact on HIV prevalence; difference in districts where Avert is the only donor￾supported project 2. What have been the challenges? How did they affect performance? How were the challenges managed? Probe – co-ordination issues; 3. What contribution has the Avert Project made to the National AIDS Control Programme? Probe – models [district comprehensive] /areas [migrants etc] adopted 4. Has Avert had a focus on: i. quality of service delivery? What have been the results? ii. capacity development? What have been the results? 5. How effective was the TSU in supporting SACS to scale-up and improve the quality of TI programs? 6. With the benefit of hindsight, what should Avert have done differently? 7. What have been the advantages and disadvantages of the Avert Society model? Probe – governance structure, governance processes 8. After June 2011, when the district programs have been transitioned to MSACS, which aspects of the Avert Project will be sustained locally? 9. What is NACO perspective on where donor support should focus in the future? AVERT End of Project Evaluation XXII 2. KEY INFORMANT INTERVIEW QUESTIONS USAID AVERT MANAGER 1. What, from a USAID perspective, has the Avert Project achieved? Probe: impact on HIV prevalence; difference in districts where Avert is the only donor￾supported project 2. What from USAID perspective have been the challenges? How did these affect performance? How were the challenges managed? Probe – co-ordination issues 3. How has USAID monitored Avert Project performance during the life of the project? 4. Has Avert had a focus on i. quality of service delivery? What have been the results? ii. capacity development? What have been the results? 5. What have been the advantages and disadvantages of the Avert Society model? Probe – governance structure, governance processes 6. With the benefit of hindsight, what should Avert have done differently? 7. After June 2011, when the district programs have been transitioned to MSACS, which aspects of the Avert Project will be sustained locally? AVERT End of Project Evaluation XXIII 3. KEY INFORMANT INTERVIEW QUESTIONS AVERT BOARD CHAIRPERSON 1. What from the Board perspective has the Avert Project achieved? Probe: impact on HIV prevalence; difference in districts where Avert is the only donor￾supported project 2. What from Board perspective have been the challenges? How did these affect performance? How were the challenges managed? Probe – coordination issues, management issues 3. How has Board monitored Avert Project performance during the life of the project? 4. Has Avert had a focus on: i. quality of service delivery? What have been the results? ii. capacity development? What have been the results? 5. What have been the advantages and disadvantages of the Avert Society model? Probe – governance structure, governance processes, 6. With the benefit of hindsight, what should Avert have done differently? 7. After June 2011, when the district programs have been transitioned to MSACS, which aspects of the Avert Project will be sustained locally? 8. What is your perspective on where donor support should focus in the future? AVERT End of Project Evaluation XXV 4. KEY INFORMANT INTERVIEW QUESTIONS MAHARASHTRA STATE AIDS CONTROL SOCIETY 1. What, from a MSACS perspective, has the Avert Project achieved? Probe: impact on HIV prevalence; difference in districts where Avert is the only donor￾supported project 2. What from MSACS’ perspective have been the challenges? How did these affect performance? How were the challenges managed? Probe – coordination issues, management issues 3. How has Avert Project coordinated with MSACS? 4. How effective was the TSU in supporting SACS to scale-up and improve the quality of TI programs? 5. How has MSACS monitored Avert Project performance during the life of the project? 6. Has Avert had a focus on: i. quality of service delivery? What have been the results? ii. capacity development? What have been the results? 7. What have been the advantages and disadvantages of the Avert Society model? Probe – governance structure, governance processes, 8. With the benefit of hindsight, what should Avert have done differently? 9. After June 2011, when the district programs have been transitioned to MSACS, which aspects of the Avert Project will be sustained locally? 10. What is your perspective on where donor support should focus in the future? AVERT End of Project Evaluation XXVI 5. KEY INFORMANT INTERVIEW QUESTIONS MUMBAI DISTRICT AIDS CONTROL SOCIETY 1. What, from MDACS’ perspective has the Avert Project achieved? 2. How has the Avert Project coordinated with MDACS? 3. How effective is the TSU in supporting MDACS to scale-up and improve the quality of TI programs? AVERT End of Project Evaluation XXVII 6. KEY INFORMANT INTERVIEW QUESTIONS FHI [BMGF] 1. What were the BMGF project activities in Thane? Was there any collaboration with the Avert Project? 2. How did the Avert Project co-ordinate with BMGF project and other donor￾supported projects in Thane? 3. What is your experience of the advantages and disadvantages of the Avert Society model? 4. How successful has the targeted intervention program been in increasing access to prevention, care and support, treatment services? 5. How did FHI/BMGF project manage the multiple and changing policy directives from NACO? 6. As district programs are transitioned to MSACS, what is FHI perspective on where donor support should focus in the future? AVERT End of Project Evaluation XXVIII 7. KEY INFORMANT INTERVIEW QUESTIONS PATHFINDER 1. Was there any collaboration between Pathfinder with the Avert Project? 2. How did the Avert Project co-ordinate with the Pathfinder Project and other donor-supported projects in Thane? 3. What is your experience of the advantages and disadvantages of the Avert Society model? 4. How successful has the targeted intervention program been in increasing access to prevention, care and support, treatment services? 5. How did the Pathfinder Project manage the multiple and changing policy directives from NACO? 6. As district programs are transitioned to MSACS, what is Pathfinder perspective on where donor support should focus in the future? AVERT End of Project Evaluation XXIX 8. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY PROJECT DIRECTOR & ASSOCIATE PROJECT DIRECTOR 1. How effective is the Avert Society model in implementing the Avert Project? Probe: the governance structure/governance processes/implementation arrangements/management systems 2. What are the fundamental/structural issues and challenges that have affected the progress of the Avert Project? 3. What difference has the Avert Project made in access to and utilization of services for MARPs, changing risk behaviors, and to HIV prevalence? 4. How successful has Avert been in developing community-based organizations and empowering MARPs to access services? 5. What affect has the Avert Project had on quality of services for MARPs. 6. What has been the role and achievement of the Avert Project in capacity building? Probe: NGO Capacity/ TSUs/STRC 7. How effective was the TSU in supporting SACS to scale-up and improve the quality of TI programs? 8. How successful is the District Comprehensive Prevention, Care, Support and Treatment program in scaling-up access to quality prevention and care services by MARPs and their partners? What have been the challenges? How were they managed? 9. How successful is the integrated care model providing care and treatment services to adult and children infected with and affected by HIV and AIDS? What have been the challenges? How were they managed? 10. Is the Link Worker Program making progress towards achieving its intended goals, objectives and outcomes? 11. How does the Avert Project coordinate with MSACS and other donor-supported projects in Maharashtra? What have been the challenges? How have you managed them? What have been particular successes of the coordination? 12. As district programs are transitioned to MSACS, what aspects of the Avert Project will be sustained locally? 13. What are your perspectives on where donor support should focus in the future? 14. Who in Avert is responsible for Ngo selection and sub-granting? 15. Who in Avert is responsible for building capacity of Network NGOs to provide targeted interventions for MARPs. AVERT End of Project Evaluation XXX 9. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON M&E 1. Please give an overview of the Avert Project M&E system : processes, tools, and reporting (on what and to whom). Probe, if needed, on DQA 2. How does the Avert Project measure its performance? What data does Avert Society hold on targets and achievements by FY? 3. How does the Avert Project measure and monitor quality of services? How has service quality changed over the last 4-5 years? 4. In addition to the BSS, how does the Avert Project measure and report consistent condom use and other behavior change by MARPs? 5. Is consistent condom use data available for all MARPs, or only FSW? 6. How have Avert Society leadership and management systems affected overall performance of the Avert Project? 7. How is the Avert Project M&E data used to improve programs? Probe: evidence-based programming feedback to NGOs AVERT End of Project Evaluation XXXI 10. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON PROGRAM Care & Support Specialist TI Specialist Communication Specialist 1. How effective has the Avert Project been? What have you achieved/what are you most proud of? Prompt: Coverage of MARPs, quality of services, increasing consistent condom usage among MARPs, HIV prevalence 2. How successful has Avert been in developing community-based organizations and empowering MARPs to access services? 3. Tell us about the success of the program interventions and models for prevention and care developed by the Avert Project. d) District Comprehensive Prevention, Care, Support and Treatment program in scaling-up access to quality prevention and care services by MARPs and their partners e) Integrated care model providing care and treatment services to adult and children infected and affected by HIV/AIDS f) HIV prevention for migrant populations and, g) Is the Link Workers Program making progress towards achieving its intended goals, objectives and outcomes? Prompt for each: What have been the challenges? How were they managed? Prompt for b) What has been the challenge for integrating adult and child care? How was this managed Prompt for c) How do you link seasonal migrants to services in their home states or districts? 4. How has evidence-based programming contributed? Please give examples of where evidence-based programming has improved outcomes. 5. Who in Avert oversees/monitors QA/QI? What efforts have been made to improve the quality of services? Are there training modules for QA/QI for NGO staff and or technical staff? 6. How has the Avert Project managed directions from NACO that were not in line with workplans. 7. What coordination is there between Avert and MSACS? How are referrals to government services (ICTC and ART centers) managed? 8. How have Avert Society leadership and management systems affected overall performance of the Avert Project? AVERT End of Project Evaluation XXXII 9. Tell us about the communications approach and materials that you are using. How has this changed since 2006? Have members of the MARP communities been involved in developing the behavior change messages and materials? 10. Tell us about the targeted intervention (TI) approach -- the successes and the challenges. 11. Tell us about the Avert Project process for NGO selection? What is unique about it? 12. Is there anything further you need to tell us about successes and challenges in care and support? AVERT End of Project Evaluation XXXIII 11. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY TSU STAFF 1. What is the purpose of the TSU? 2. How is the TSU involved in capacity building? 3. How effective is the TSU in supporting SACS to scale-up and improve the quality of TI programs? Please give examples? 4. What formal interaction do you have with the Avert Project staff? On what issues do you coordinate? 5. Who does team leader TSU report to? 6. Who in TSU oversees/monitors QA/QI? What efforts have been made to improve the quality of services? Are there training modules for QA/QI for NGO staff and/or technical staff? 7. What is Avert’s role in managing the TSU? AVERT End of Project Evaluation XXXIV 12. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON NGO CAPACITY BUILDING 1. What capacity building does Avert undertake for network NGOs? 2. In addition to core skills in targeted interventions for MARPs, do you build NGO capacity in: i. Administration/management ii. Financial management iii. Monitoring and evaluation iv. Leadership v. Planning vi. Quality Assurance 3. What is the Avert methodology for capacity building? What more than training is involved? 4. How do you ensure the quality and effectiveness of training? How do you identify training needs and, later, gaps in training? AVERT End of Project Evaluation XXXV 13. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY ON NGO SUB-GRANTING 1. Tell us about the Avert Project process for NGO selection. What is unique about it? 2. Describe the Avert process for sub-granting to network NGOs. Prompt: do you use contracts or MOUs? For how long are the agreements with each NGO 4. What work planning does Avert do with its NGOs? 5. What reporting is required of the NGOs? 6. How does Avert identify poorly performing NGOs? What actions do you take to improve NGO performance? 7. What challenges does Avert face with sub-grant disbursements? How are these managed? AVERT End of Project Evaluation XXXVI 14. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY FOR GOVERNMENT OFFICERS IN THE FIELD Please record the district, and the government officer’s name and position 1. What has the Avert Society/[name local NGO] achieved in your district? 2. What have been the challenges for collaboration with the Avert Society/[name local NGO]? 3. How do you coordinate the Avert Society/[name local NGO] program with government programs in your district? 4. What will continue after the hand over of donor-supported programs to the Government of Maharashtra? AVERT End of Project Evaluation XXXVII 15. KEY INFORMANT INTERVIEW QUESTIONS AVERT SOCIETY NGO STAFF Please record the district, the name of the NGO, the Avert program[s] being implemented, and the population being served as well as the name and position of the respondent. 1. How long has the NGO been implementing the Avert program? 2. What grade (B, B+, A, A+ etc) has Avert given the NGO? 3. How has the grade changed? 4. What assistance has Avert provided? Probe: training? System strengthening? On-the-job coaching and mentoring? Supportive supervision? Please record examples. 5. How have you changed the way you manage and deliver services for your community while working with Avert? Probe: How have services improved? Please record examples. 6. What have you achieved? What are you most proud of? 7. What have been the challenges and how have you managed them? AVERT End of Project Evaluation XXXVIII 16. BENEFICIARY GROUP DISCUSSION AREAS: After introductions, ask if the beneficiaries are comfortable talking with us. Assure them that anything they say will not be attributed to them personally in our reports. All comments will be recorded will be anonymously. Record the type of beneficiaries. 1. Tell us about the [NGO] services you receive. 2. How have the services benefited you and others in the community? 3. For how long have you benefited from the services? 4. Have you formed a self-help group? If yes, a) How did you do that? b) How does the self help group benefit you and the other members? 5. Tell us about any problems there have been with the [NGO] services. Probe: what did you do about these problems? 6. Is there anything else you want us to know? AVERT End of Project Evaluation XXXIX ANNEX D: EVALUATION CALENDAR AVERT End of Project Evaluation XLI ANNEX E: CORRECT AND CONSISTENT CONDOM USE BROTHEL BASED FSWS (BB-FSWS) Correct and Consistent condom use with paying partners Districts Wave II 2004-05 Wave III 2005-06 Wave IV 2007-08 Wave V 2009 Aurangabad Not Available as only NBB-FSWs were covered Nagpur 64 77 91 95.6 Solapur 50 98 91 93.45 Thane 85 72 80 93.65 Correct and Consistent condom use with non paying partners Districts Wave II Wave III Wave IV Wave V Aurangabad Not Available as only NBB-FSWs were covered Nagpur 3 19 35 56.1 Solapur 5 15 34 26.7 Thane 4 30 27 80.2* * Seems like an outlier as it is very high NON BROTHEL BASED FSWS (NBB-FSWS) Correct and Consistent condom use with paying partners Districts Wave II 2004-05 Wave III 2005-06 Wave IV 2007-08 Wave V 2009 Aurangabad 46 86 69 88.25 Nagpur Not Available as only BB FSWs were covered 93.55 Solapur 57 98 79 98.25 Thane 50 40 80 92.65 Correct and Consistent condom use with non paying partners Districts Wave II Wave III Wave IV Wave V Aurangabad 19 8 8 18.1 Nagpur Not Available as only BB FSWs were covered 50 Solapur 16 5 20 37.1 Thane 10 13 16 23.7 AVERT End of Project Evaluation XLII MEN WHO HAVE SEX WITH MEN Correct and Consistent condom use with commercial male partner Districts Wave II Wave III Wave IV Wave V Aurangabad & Nagpur 41 43 78 Aurangabad In previous waves the estimates for Aurangabad & Nagpur were combined 79 Nagpur 93.1 Solapur 74 100 87 89 Thane 69 78 59 73 Correct and Consistent condom use with non commercial partners Districts Wave II Wave III Wave IV Wave V Aurangabad & Nagpur 37 36 83 Aurangabad In previous waves the estimates for Aurangabad & Nagpur were combined 78.6 Nagpur 88.6 Solapur 73 95 81 89 Thane 56 62 59 66.5 AVERT End of Project Evaluation XLIII ANNEX F: HIV PREVALENCE IN AVERT PRIORITY DISTRICTS ANC PREVALENCE DISTRICT 2002 2003 2004 2005 2006 2007 2008 Aurangabad 1.25 0.25 0.25 0.00 1.25 0.50 0.25 Jalna 0.00 0.25 1.00 1.25 0.75 0.50 0.38 Nagpur 2.25 2.75 1.25 1.50 0.50 1.25 0.63 Solapur 0.00 2.00 2.00 2.75 0.50 1.50 1.50 Thane 2.00 4.25 1.50 2.00 0.75 1.75 0.75 FSW PREVALENCE Aurangabad 2.00 Nagpur 17.20 Solapur 6.40 Thane 38.00 25.20 28.40 32.40 HIV PREVALENCE AT STI SITES Aurangabad 7.60 14.00 5.60 6.80 10.00 10.40 7.60 Nagpur 21.20 22.00 18.80 20.40 20.40 13.60 13.60 Thane 8.30 14.40 4.00 8.00 6.00 7.20 12.00 SOURCE: HSS 2002-2008 AVERT End of Project Evaluation XLV ANNEX G: IMPROVED PE/VCL PERFORMANCE FOLLOWING INTRODUCTION OF NEW NACO TRACKING FORMATS AND RETRAINING IN BCC Humsaaya (MSM) - June 2010 Table 13: Peer Educator No Activities No. of Activities Persons Male Female TG Total 1 No. of Persons contacted by PE 1313 1449 0 161 1610 2 No. of one to one interaction (Follow up visits) 1248 1099 0 149 1248 3 No. of one to group interaction (Follow up visits) 65 350 0 12 362 4 No. of IEC material distributed 274 236 0 19 255 5 No. of STD cases referred 59 56 0 3 59 6 No. of STD cases referred by PE taken for Treatment 0 0 0 0 0 7 No. of condom distributed (Free) 658 3063 0 390 3453 8 No. of condom distributed (Male condom Sale) 0 0 0 0 0 No. of condom distributed (Female condom Sale) 0 0 0 0 0 9 No. of condom demonstrations 217 318 0 23 341 10 No. of condom re-demonstrations 166 266 0 10 276 11 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation XLVI Humsaaya (MSM) - Feb. 2011 Table 13: Peer Educators No Activities No. of Activities Persons Male Female TG Total 1 No of Persons contacted by PE 1937 2090 0 185 2275 2 No. of one to one interaction (Follow up visits) 1875 1711 0 164 1875 3 No. of one to group interaction (Follow up visits) 62 379 0 21 400 4 No. of IEC material distributed 327 327 0 0 327 5 No. of STD cases suspected & referred 117 107 0 10 117 6 No. of STD cases referred by PE taken for Treatment 1 1 0 0 1 7 No. of condom distributed (Free) 1901 9042 0 1032 10074 8 No. of condom distributed (Male condom Sale) 0 0 0 0 0 No. of condom distributed (Female condom Sale) 0 0 0 0 0 9 No. of condom demonstrations 270 335 0 22 357 10 No. of condom re-demonstrations 255 304 0 19 323 11 Any other activity 0 0 0 0 0 Source: Avert Project Database AVERT End of Project Evaluation XLVII Sankalp (IDU) -- June 2010 Table 13: Peer Educator No Activities No. of Activities Persons Male Female TG Total 1 No. of Persons contacted by PE 0 0 555 2 No. of one to one interaction (Follow up visits) 321 321 0 0 321 3 No. of one to group interaction (Follow up visits) 66 234 0 0 234 4 No. of IEC material distributed 2 67 43 0 110 5 No. of STD cases referred 0 0 0 0 0 6 No. of STD cases referred by PE taken for Treatment 0 0 0 0 0 7 No. of condom distributed (Free) 0 80 0 0 80 8 No. of condom distributed (Male condom Sale) 0 0 0 0 0 No. of condom distributed (Female condom Sale) 0 0 0 0 0 9 No. of condom demonstrations 0 0 0 0 0 10 No. of condom re-demonstrations 0 0 0 0 0 11 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation XLVIII Sankalp (IDU) - Feb. 2011 Table 13: Vol. Peer Leader No. Activities No. of Activities Persons Male Female TG Total 1 No. of Persons contacted by PE 1351 1874 1 0 1875 2 No. of one to one interaction (Follow up visits) 1092 1092 0 1092 3 No. of one to group interaction (Follow up visits) 259 782 1 0 783 4 No. of IEC material distributed 0 526 1 0 527 5 No. of STD cases suspected & referred 0 0 0 0 0 6 No. of STD cases referred by PE taken for Treatment 0 0 0 0 0 7 No. of condom distributed (Free) 0 118 0 0 118 8 No. of condom distributed (Male condom Sale) 0 0 0 0 0 No. of condom distributed (Female condom Sale) 0 0 0 0 0 9 No. of condom demonstrations 0 0 0 0 0 10 No. of condom re-demonstrations 0 0 0 0 0 11 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation XLIX BAJSS (Migrant) - May 2010 Activities No. of Activities Persons Male Female TG Total No. of Persons by PE 626 981 85 0 1066 No. of one to one interaction 508 456 42 0 498 No. of one to group interaction 118 525 43 0 568 No. if IEC material distributed 110 110 0 0 110 No. of STD cases referred 55 55 0 0 55 No. of STD cases referred by PE taken for treatment 0 0 0 0 0 No. of condom distributed (free) 0 0 0 0 0 No. of condom distributed (sale) 112 435 0 0 435 No. of condom demonstrations 282 682 45 0 727 No. of condom re-demonstrations 296 279 17 0 296 AVERT End of Project Evaluation L BAJSS (Migrant) -Feb 2011 Table 13: Vol. Peer Leader Activities No. of Activities Persons Male Female TG Total No. of Persons contacted by PE 1189 1766 134 0 1900 No. of one to one interaction (Follow up visits) 981 913 68 0 981 No. of one to group interaction (Follow up visits) 208 853 66 0 919 No. of IEC material distributed 1171 1291 146 0 1437 No. of STD cases suspected & referred 68 51 17 0 68 No. of STD cases referred by PE taken for Treatment 12 11 1 0 12 No. of condom distributed (Free) 1021 9925 415 0 10340 No. of condom distributed (Male condom Sale) 4 20 0 0 20 No. of condom distributed (Female condom Sale) 0 0 0 0 0 No. of condom demonstrations 1032 1950 113 0 2063 No. of condom re-demonstrations 1032 981 65 0 1046 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation LI MGVS (FSW) - May 2010 Table 13: Peer Educator Activities No. of Activities Persons Male Female TG Total No. of Persons contacted by PE 1173 57 1370 0 1427 No. of one to one interaction (Follow up visits) 1069 57 1012 0 1069 No. of one to group interaction (Follow up visits) 104 0 358 0 358 No. of IEC material distributed 100 0 100 0 100 No. of STD cases referred 310 0 310 0 310 No. of STD cases referred by PE taken for Treatment 252 0 252 0 252 No. of condom distributed (Free) 1012 0 58193 0 58193 No. of condom distributed (Male condom Sale) 26 0 1890 0 1890 No. of condom distributed (Female condom Sale) 0 0 0 No. of condom demonstrations 197 0 197 0 197 No. of condom re-demonstrations 110 0 110 0 110 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation LII MGVS (FSW)- Feb. 2011 Table 13: Peer Educators Activities No. of Activities Persons Male Female TG Total No. of Persons contacted by PE 1458 0 1895 0 1895 No. of one to one interaction (Follow up visits) 1285 0 1285 0 1285 No. of one to group interaction (Follow up visits) 173 0 610 0 610 No. of IEC material distributed 0 0 400 0 400 No. of STD cases suspected & referred 460 0 460 0 460 No. of STD cases referred by PE taken for Treatment 0 0 0 0 0 No. of condom distributed (Free) 1009 0 38609 0 38609 No. of condom distributed (Male condom Sale) 0 0 0 0 0 No. of condom distributed (Female condom Sale) 13 0 1210 0 1210 No. of condom demonstrations 397 0 478 0 478 No. of condom re-demonstrations 393 0 415 0 415 Any other activity 0 0 0 0 0 AVERT End of Project Evaluation LIII ANNEX H: COMPARISON OF PERFORMANCE IN AVERT DISTRICTS TO NON-AVERT DISTRICTS Comparison of Aurangabad with Parbhani to assess the Outcome/Impact Note: (1) Parbhani has been selected for comparison with Aurangabad as most of the social and geographic characteristics are similar to that of Aurangabad. Moreover, it also belongs to the same administrative division i.e. Aurangabad Division (2) Only HIV Testing for the district and Prevalence data are available over the years and hence a trend of the same is provided. As BSS data for Parbhani is available for only one time point, 2009 the graph for Behavioural Indicators is not provided (3) The number of ICTCs were increased in 2007 and hence the data has been provided from 2007 onwards. Prevalence ANC 2004 2005 2006 2007 2008 Aurangabad 0.25 0 1.25 0.5 0.25 Parbhani 0.50 0.50 0.25 1.25 0.63 ANC Rural 2004 2005 2006 2007 2008 Aurangabad 0 0 0.25 0 0.25 Parbhani 1.00 0.50 1.50 1.00 0.25 FSW 2004 2005 2006 2007 2008 Aurangabad 2 Parbhani 15.60 10.40 8.80 4.40 Source: HSS 2004, 2005, 2006, 2007, 2008 AVERT End of Project Evaluation LIV HIV Testing Total Nos. Tested 2007 2008 2009 2010 Aurangabad 36064 35715 64935 46026 Parbhani 15387 14405 27003 31184 Nos. Tested Positive 2007 2008 2009 2010 Aurangabad 2448 1735 1652 1500 Parbhani 1089 1005 1150 1116 Positivity Rate 2007 2008 2009 2010 Aurangabad 6.79 4.86 2.54 3.26 Parbhani 7.08 6.98 4.26 3.58 Source: ICTC Records 2007, 2008, 2009, 2010, Avert database AVERT End of Project Evaluation LV Behaviour Indicators FSW - CCCU with paying partners 2004-05 2005-06 2007-08 2009 Aurangabad 46 86 69 97.7 Parbhani 77.95 FSW - CCCU with non paying partners 2004-05 2005-06 2007-08 2009 Aurangabad 19 8 8 18.1 Parbhani 35.6 CCCU - Correct and Consistent Condom Use Source: BSS Wave II, III, IV, V AVERT End of Project Evaluation LVI Comparison of Nagpur with Chandrapur to assess the Outcome/Impact Note: (1) Chandrapur has been selected for comparison with Nagpur as most of the social and geographic characteristics are similar to that of Nagpur. Moreover, it also belongs to the same administrative division i.e. Nagpur Division (2) Only HIV Testing for the district and Prevalence data are available over the years and hence a trend of the same is provided. As BSS data for Chandrapur is available for only one time point, 2009 the graph for Behavioural Indicators is not provided (3) The number of ICTCs were increased in 2007 and hence the data has been provided from 2007 onwards. Prevalence ANC 2004 2005 2006 2007 2008 Nagpur 1.25 1.5 0.5 1.25 0.63 Chandrapur 3 3.5 3.5 2 1.51 ANC Rural 2004 2005 2006 2007 2008 Nagpur 0 0 0.25 0 0.25 Chandrapur 1.00 0.50 1.50 1.00 0.25 FSW 2004 2005 2006 2007 2008 Nagpur 17.2 Chandrapur 22 20.8 12.8 8.4 Source: HSS 2004, 2005, 2006, 2007, 2008 AVERT End of Project Evaluation LVII HIV Testing Total Nos. Tested 2007 2008 2009 2010 Nagpur 76883 76008 100411 65505 Chandrapur 17333 21922 52510 42440 Nos. Tested Positive 2007 2008 2009 2010 Nagpur 8865 6005 3397 3020 Chandrapur 1008 1202 1237 1260 Positivity Rate 2007 2008 2009 2010 Nagpur 11.53 7.90 3.38 4.61 Chandrapur 5.82 5.48 2.36 2.97 Source: ICTC Records 2007, 2008, 2009, 2010 AVERT End of Project Evaluation LVIII Behaviour Indicators FSW - CCCU with paying partners 2004-05 2005-06 2007-08 2009 Nagpur 64 77 91 95.6 Chandrapur 98.7 FSW - CCCU with non paying partners 2004-05 2005-06 2007-08 2009 Nagpur 3 19 35 56.1 Chandrapur 31.1 CCCU - Correct and Consistent Condom Use Source: BSS Wave II, III, IV, V AVERT End of Project Evaluation LIX ANNEX I: AVERT SOCIETY’S PROJECT DIRECTORS S. No. Name Period No. of months Interim or Full time 1. Mr. G. Manoj Aug. 11, 2001 until Sept. 4, 2001 < 1 Interim 2. Mr. K. Vaidyanathan Sept. 5, 2001 until Sept. 4, 2003 24 Full Time Recruited from open market 3. Mr. Sanjay Sawant Sept. 5 2003 until Oct. 21, 2004 14 Interim, Addl. Charge of PD, MSACS 4. Dr. N.J. Rathod Oct. 21, 2004 until Mar.13, 2005 5 Interim, Addl. Charge of Jt. Director, NPCB 5. Dr. S.M. Sapatnekar Mar.14, 2005 until Mar.12, 2007 24 Full time Recruited from open market 6. Ms. Anna Joy Mar. 28, 2007 until June 3, 2007 2 Addl. Charge of Associate PD, Avert 7. Ms Anna Joy/ Dr. N.J. Rathod June 4, 2007 until April 24, 2008 11 Interim, Addl. Charge of Jt. Director, NPCB; Dr Rathod also nominated as PD. 8. Ms. Anna Joy April 24, 2008 until July, 2010 28 Addl. Charge of Associate PD, Avert 9. Ms. Smriti Acharya July, 2010 until to date 9 Full Time AVERT End of Project Evaluation LX