LESSONS LEARNED FROM THE BÉNIN HIV/AIDS PREVENTION PROGRAM (BHAPP) FINAL EVALUATION OF BHAPP, AS IMPLEMENTED BY AFRICARE IN COLLABORATION WITH JHPIEGO 2002 - 2006 JUNE 2006 This publication was produced for review by the United States Agency for International Development. It was prepared by Initiatives Inc. BHAPP FINAL EVALUATION REPORT, JUNE 2006 i ACKNOWLEDGEMENTS The undersigned, contracted by Initiatives, Inc. to conduct the final evaluation of the Bénin HIV/AIDS Prevention Program on behalf of USAID/Bénin, wish to express deep appreciation to BHAPP Chief of Party Mbella Ngongi and staff for their help and their responsiveness throughout the assignment. Considering that BHAPP was to permanently close its doors only days after the evaluation was concluded, it would have been understandable had the staff been distracted by worries about their future. That they were able to rise above these concerns and give willingly of their time to answer our many questions, as well as to accompany us on all of our field visits, spoke volumes for their professionalism and pride in their work. We wish them all well, and hope this report reflects the devotion they gave to the project and the lessons learned from it. Among the many stakeholders with whom we met, we want especially to thank Dr. Marcel Zannou and Dr. Edgar Lafia, Coordonnateur and Coordonnateur Adjoint of PNLS, and Dr. Mèdégan Valentine, Secrétaire Permanent of CNLS, for their frank and helpful comments about the impact of BHAPP and their hopes for the future. We also thank the many NGO and health service partners with whom we met sûr le terrain. They too were grappling with the implications of cessation of BHAPP support, but were anxious to share their experiences and ideas with us, in the process illustrating their deep commitment to preventing the spread of HIV and sexually transmitted infections in Bénin. We are grateful for the assistance and wise counsel provided us throughout the evaluation by Don Dickerson and the other members of the USAID/Bénin Family Health Team, and by Abdalla Meftuh, Bénin Country Representative for Africare. Finally, we warmly thank Nicole Dupré, Initiatives’ Office Manager, for managing this assignment with a steady hand from beginning to end. Dr. Léon K. Kessou Eliot T. Putnam, Jr. June 2006 BHAPP FINAL EVALUATION REPORT, JUNE 2006 ii LESSONS LEARNED FROM THE BÉNIN HIV/AIDS PREVENTION PROGRAM (BHAPP) FINAL EVALUATION OF BHAPP, AS IMPLEMENTED BY AFRICARE IN COLLABORATION WITH JHPIEGO 2002 - 2006 ELIOT T. PUTNAM, JR. DR. LÉON K. KESSOU The information contained in this report is the product of an evaluation conducted by Initiatives Inc. The report was prepared under the auspices of the Technical Assistance and Support Contract (TASC2 Global Health), implemented by Initiatives Inc. under Task Order No. GHS-I-83-03-00040-00 issued by the U.S. Agency for International Development. Initiatives Inc. 376 Boylston Street, Suite 4C Boston MA 02116 USA TEL (617) 262-0293 FAX (617) 262-2514 www.initiativesinc.com DISCLAIMER The opinions expressed in this report are those of the individual authors and do not necessarily represent the views of the United States Agency for International Development or the United States Government. BHAPP FINAL EVALUATION REPORT, JUNE 2006 iii ACRONYMS ARV.....................Anti-retroviral drug BCC ....................Behavior change communication BHAPP................Bénin HIV/AIDS Prevention Program BSS I, II................First and Second Behavioral Surveillance Surveys CCC....................Communication pour changement de comportement (BCC) CIPEC .................Centre d’Information, de Prospective et de Conseil (departmental unit of PNLS) CNLS ..................Comité National de Lutte contre le SIDA (National Committee for the battle vs. AIDS) CS ........................Centre de santé (health center) CSA.....................Centre de santé d’arrondissement (local health center) CSM.....................Contraceptive social marketing DDSP ..................Direction Départementale de la Santé Publique ESDGB................Enquête de surveillance de 2ème génération (BSS II) FHT .....................Family Health Team (of USAID/Bénin) GOB....................Government of Bénin IST .......................Infection sexuellement transmissible (STI) MAP.....................Multisectoral AIDS Program of the World Bank MDM...................Médecins du Monde (Doctors of the World) MOH ..................Ministry of Health NGO...................Non-governmental organization ONG...................Organisation non-gouvernementale (NGO) PIVA ....................Projet Intégré de VIH Ahéme (integrated HIV project of MDM) PLWHA..............Persons living with HIV and AIDS PMP .....................Performance monitoring plan (of BHAPP) PMTCT...............Prevention of mother-to-child transmission of HIV PNLS ...................Programme National de Lutte contre le SIDA (National Program for the battle vs. AIDS) (a unit of the MOH) PPLS ....................Programme plurisectoriel de lutte contre le SIDA (of World Bank) POSS ...................Plan operationel de secteur santé PSI........................Population Services International PTME...................Prévention de transmission VIH de mère á enfant (PMTCT) PPLS ....................Projet Plurisectoriel de Lutte contre le SIDA (World Bank/MAP) PVVIH .................Personnes vivant avec VIH/SIDA (PLWHA) RFA......................Request for applications SIDA....................AIDS SIDA 3 ................Canadian funded AIDS and STI prevention and treatment project STI .......................Sexually transmitted infection TA........................Technical assistance TOT ....................Training of trainers VCT.....................Voluntary counseling and testing VIH ......................HIV BHAPP FINAL EVALUATION REPORT, JUNE 2006 iv CONTENTS ACKNOWLEDGEMENTS................................................................................................................................................i ACRONYMS.......................................................................................................................................................................iii EXECUTIVE SUMMARY.................................................................................................................................................. 1 I. INTRODUCTION......................................................................................................................................................... 3 I.A. THE RESPONSE TO HIV AND AIDS IN BÉNIN...............................................................................3 I.B. THE BHAPP APPROACH ........................................................................................................................3 I.C. OBJECTIVE OF THIS EVALUATION....................................................................................................4 II. METHODOLOGY........................................................................................................................................................ 5 III. FINDINGS AND LESSONS LEARNED.................................................................................................................. 6 III.A. PROJECT DESIGN AND IMPACT – CENTRAL LEVEL...................................................................6 III.B. PROJECT DESIGN AND IMPACT – REGIONAL AND COMMUNITY LEVEL ........................8 III.C. OTHER IMPLICATIONS / RECOMMENDATIONS FOR USAID-FUNDED ACTIVITIES...............................................................................................................12 IV. IMPACT OF BHAPP VS. PERFORMANCE MONITORING PLAN (PMP).................................................13 V. COLLABORATION WITH COUNTERPARTS AND STAKEHOLDERS ...................................................14 VI. PROJECT MANAGMENT.......................................................................................................................................15 VII. CONCLUSION ........................................................................................................................................................16 ANNEXES 1. PERSONS MET AND SITES VISITED (in French) 2. BHAPP RESULTS vs. TARGETS AND INDICATORS 3. EVALUATION SCOPE OF WORK BHAPP FINAL EVALUATION REPORT, JUNE 2006 1 EXECUTIVE SUMMARY A key element of USAID's support for efforts to stem troubling increases in the rate of HIV transmission and sexually transmitted infections (STI) and the spread of AIDS in Bénin has been the Bénin HIV/AIDS Prevention Program, or BHAPP. Funded under a $4.5 million, four-year (2002 - 2006) Cooperative Agreement (CA) between USAID and Africare, in partnership with JHPIEGO, BHAPP was designed to reinforce the capacity of the Ministry of Health's Programme National de Lutte contre le SIDA (PNLS) to coordinate and manage a nationwide program of HIV/AIDS prevention. With the conclusion of the BHAPP CA in May 2006, it was deemed essential to the prospects for future such interventions in Bénin to conduct a final evaluation of BHAPP's work and assess its impact at central and decentralized levels. Coming at project's end as it did, this evaluation was approached by its two-person evaluation team less as a traditional exercise in dissecting strengths and weaknesses, and more as one of synthesizing lessons learned from four years of activity. In this way, the team hoped, the accomplishments of the project, as well as the things it was not able to achieve, would have the best chance of informing the work that will go forward, notably through a new USAID￾funded HIV and AIDS-focused initiative. The team sought these lessons, which are highlighted throughout the report that follows, through extensive meetings with BHAPP, PNLS and other national and international stakeholders, and through contacts with NGO partners and health service providers in the field who had benefited from BHAPP interventions. The team found that BHAPP played an important role in furthering Bénin’s national effort to prevent the spread of HIV, enhance management of sexually transmitted infections, and strengthen the overall policy environment for dealing with the HIV/AIDS epidemic. In its central mission of providing the technical assistance necessary to strengthening the capacity of the PNLS to lead the national prevention effort, it left a legacy of high quality norms and standards for BCC, STI management and other key technical areas. It also left an agency technically strengthened at the center, and appreciative of that strengthening. But much remains to be done, especially in strengthening linkages between the central PNLS and its regional units for the sharing and support of successes and innovations in the field. In the field, BHAPP launched a creative BCC initiative through the intermediary of micro￾projects managed by indigenous NGOs. While the team found the micro-project structure to have its limitations, it resulted in training and fielding of over 1,500 peer educators equipped to reach out with prevention and treatment messages to key groups in the community at high risk of infection. The project was less successful in reinforcing condom social marketing (CSM) activities through its outreach and communication training. In 23 selected health facilities in the zones sanitaires where it was active, BHAPP also succeeded in demonstrably strengthening the capacity of clinical personnel to diagnose and treat STIs at a level of service quality well above that previously available. Project inputs, notably development of a technically precise algorithm for STI management, and the development of and training in the concept of supervision facilitante, were widely praised and appreciated, and have set a high standard for national replication. Full coverage of target zones sanitaires, which might have been achieved by reducing their number, would have further enhanced the impact of this important element of the BHAPP project. At a national level, BHAPP contributed significantly to improvement of the policy environment for a multisectoral approach to confronting HIV. It mapped HIV/AIDS interventions of international and bilateral donors in Bénin, supported preparation of the second national behavioral surveillance survey (BSS II), and provided assistance to drafting the national HIV/AIDS law and revising the national BHAPP FINAL EVALUATION REPORT, JUNE 2006 2 strategic plan for dealing with the epidemic. In so doing, it not only pursued its primary mission of strengthening the capacity of the PNLS to lead the fight from the health perspective, but also contributed to the more overarching role of the Comité National de Lutte contre le SIDA, or CNLS. The BHAPP CA was authorized for only four years, and all project activities, other than final reporting, came to a close as of May 16, 2006. While the shortness of the project left much of its work unfinished, BHAPP left important technical accomplishments in place, and paved the way for expanded and improved efforts, working through the PNLS, to expand AIDS awareness, reduce HIV transmission and strengthen services for treatment of STIs. BHAPP FINAL EVALUATION REPORT, JUNE 2006 3 I. INTRODUCTION I.A. THE RESPONSE TO HIV AND AIDS IN BÉNIN HIV prevalence in Bénin, currently estimated at close to 2%, has yet to reach the dire levels seen in many other African countries. Nonetheless it is high enough to be considered an epidemic, and to threaten the public health system of a nation already burdened by high levels of infant mortality and morbidity, high fertility and population growth rates, and inadequate access to health services. HIV and AIDS could increase sharply without a vigorous, coordinated response, which is the mandate of the Ministry of Health's (MOH) Programme National de Lutte contre le SIDA (PNLS). Of particular concern to the PNLS are groups at high risk of infection, such as sex workers, truck and taxi drivers, out-of-school youth, and large numbers of people suffering from sexually transmitted infections (STI) that the system has been ill-equipped to handle. USAID/Bénin has long supported efforts to improve access to, quality of and demand for health services in the country, most recently through its Bénin Integrated Family Health Program. For the past four years (2002 - 2006), a key element of this program has been the Bénin HIV/AIDS Prevention Program, or BHAPP. Under a four-year, $4.5 million Cooperative Agreement between USAID and Africare, in partnership with JHPIEGO, BHAPP was designed “to reduce the rate of HIV/AIDS transmission in Bénin … (by) reinforcing the PNLS' capacity to coordinate and manage a nationwide HIV/AIDS prevention program.”1 With BHAPP activities concluding as of May 16, 2006, USAID/Bénin has worked with the MOH and other national entities to design a follow-on project including HIV prevention and services, for which an RFA was issued in May, 2006. 1 "Conceptual Approach", p. B-4, BHAPP Cooperative Agreement document, signed 5/10/02. I.B. THE BHAPP APPROACH BHAPP took a two-pronged approach to its work of building PNLS’ capacity. At a national level it worked with the PNLS on annual program planning, and provided extensive technical assistance (TA) in the development of a series of detailed norms and standards for HIV, AIDS and STI prevention and services. It mapped HIV/AIDS-related interventions of international and bilateral donors in Bénin, provided technical and logistical support to preparation of the second national behavioral surveillance survey (BSS II), and provided assistance to drafting the national HIV/AIDS law and revising the national Cadre Stratégique (Strategic Plan) for dealing with the epidemic. In these and other ways it contributed significantly to improvement of the policy environment for a multi-sectoral approach to confronting HIV. In so doing, BHAPP not only pursued its primary mission of strengthening the capacity of the PNLS to lead the fight from the health perspective, but also contributed directly and indirectly to the more overarching role of the Comité National de Lutte contre le SIDA, or CNLS. At a decentralized level, BHAPP implemented service and communication activities in selected zones sanitaires (health zones) of five of Bénin's twelve départements - Atlantique, Mono/Kouffo, and Zou/Collines. The expectation was that results of these activities would be funneled back to inform and strengthen PNLS' capacity to manage a national program. Through funding of a series of “micro-projects,” BHAPP supported behavior change communication (BCC) outreach activities of over 25 indigenous NGOs, enlisting the cooperation of elected officials and other leaders in communities where outreach took place. Through training, technical assistance and development of materials, the project strengthened the capacity of personnel of 23 health centers in different health zones to diagnose and treat STIs, and to use STI case management as a referral point for HIV testing. Finally, BHAPP was intended to reinforce condom social marketing (CSM) BHAPP FINAL EVALUATION REPORT, JUNE 2006 4 activities through its outreach and communication training. I.C. OBJECTIVE OF THIS EVALUATION The BHAPP CA was authorized for only four years, and all project activities, other than final reporting, came to a close as of May 16, 2006. Coming at the end as it did, this evaluation therefore was conceived by the evaluation team, with the concurrence of USAID/Bénin, less as an exercise in dissecting strengths and weaknesses than as one of synthesizing lessons learned from four years of activity. In this way, it is hoped, the accomplishments of the project, as well as the things it was not able to achieve, can have the best chance of impacting in a positive way the work that will go forward under a new HIV and AIDS-focused initiative. BHAPP FINAL EVALUATION REPORT, JUNE 2006 5 II. METHODOLOGY The two-person evaluation team, a Béninois physician/health economist and a U.S. consultant, convened in Cotonou on April 19, 2006. Each had previously reviewed a wide range of project documents, and would continue to do so throughout the exercise. These included (but were not limited to) quarterly and annual implementation reports, planning documents, BHAPP’s own studies and other analyses, the project's mid-term evaluation, NGO reports and proposals, and technical norms and standards developed by the project. Early on, the team determined that it needed to see project activities first hand in each of the departments, and in as many health zones as possible, where BHAPP was active. After a first round of meetings with PNLS, CNLS, the World Bank and other key stakeholders, the team spent three days visiting NGOs, health centers and community leaders in Mono/Kouffo and Zou/Collines, and thereafter did a similar one-day circuit in Atlantique. In the process it met with staffs of eight of BHAPP's NGO partners, with several médecins-chefs (doctors￾in-charge) of health centers and other health personnel, and with numerous elected officials. The team also met with the directors of two Centres d'Information, de Prospective et de Conseil, or CIPECs. These are regional units of PNLS, of which there are five nationwide. The team sought to understand how the role of CIPECs could be maximized in terms of PNLS capacity building. In all of its visits the team, at its request, was accompanied by at least one senior member of the BHAPP professional staff, which greatly enhanced the growth of our understanding of the project. Field visits enabled the team to fully understand the nature and stages of BHAPP support to NGOs for BCC micro-projects, and to health facilities for strengthening management of STI. It also enabled us to discern the sense of urgency at all levels that everything be done to prevent further spread of HIV, and concern that the discontinuation of BHAPP interventions, much appreciated on all sides, might compromise that mission. Following this exposure to the realities on the ground, the team had numerous follow-up meetings with BHAPP staff, PNLS, the USAID FHT and others before beginning to synthesize its findings, and to articulate the lessons it found to have been taught by the project. These lessons are highlighted in the report that follows. (See Annex 1 for a list of organizations and individuals contacted.) BHAPP FINAL EVALUATION REPORT, JUNE 2006 6 III. FINDINGS AND LESSONS LEARNED III.A. PROJECT DESIGN AND IMPACT – CENTRAL LEVEL As noted, the central objective of the Bénin HIV/AIDS Prevention Program was to strengthen the capacity of the PNLS to manage and coordinate health-related interventions of a nationwide program to prevent the spread of HIV and manage treatment of STIs. BHAPP sought to do this in two ways: 1. Reinforcing Technical Capacity of PNLS Key to the technical assistance offered PNLS by BHAPP was development of a series of norms, standards and procedures relating to different aspects of prevention of transmission of HIV, treatment of those infected with HIV and other STIs, and caring for those afflicted with AIDS. Through an extensive process led by BHAPP’s technical experts, and involving not only PNLS but key donors such as the World Bank, these norms, standards and procedures were drafted, reviewed, field tested, finalized and published, often in a process taking a year or two to complete. Thus, one of BHAPP’s most important legacies resides in the availability to PNLS of a series of detailed, carefully conceived technical documents to guide activities in the field. These include, for example: • conceptual framework, norms and standards for communicating the importance of changing behavior in preventing spread of HIV and STIs (BCC), which reinforced PNLS’ leadership in BCC nationwide; 2 • procedures, training guidelines, quality assurance norms and algorithms dealing with the diagnosis and management of sexually transmitted infections (STI) in health centers of zones sanitaires; 2 Stratégies, Normes, Procédures Nationales en Communication pour un Changement de Comportements, February 2004. • standards and national directives for voluntary screening and testing for HIV, and caring for people living with HIV and AIDS (PLWHA); 3 • norms, standards and procedures relating to laboratory practices in health centers that test for HIV and STIs. It is in the nature of organizations and projects to want to create their own documents and materials as new initiatives are launched. In some cases, this “reinventing the wheel” can make sense. However, in the case of the technical resources created for the PNLS with the assistance of BHAPP, the evaluation team feels strongly that these materials are ready for wide dissemination and extensive use before consideration is given to revisions or new documents. Apart from this developmental work, which of course had PNLS input, the effectiveness of the day-to-day TA/capacity building relationship between BHAPP and PNLS technical staffs was less clear. At first the expected housing of technical advisors at PNLS was not possible due to space constraints. Later, when PNLS moved to larger quarters, it was able to provide two BHAPP experts, those responsible for BCC and for STIs and epidemiological monitoring, with offices in their building. But despite this proximity, the working relationship, while cordial, was not as participatory and continuous as would have been desirable. This was true despite efforts to the contrary, including a three-day PNLS/BHAPP retreat in January of 2005 to review their working relationship. In part this seems to have stemmed from a lack of clarity between PNLS and BHAPP as to the precise meaning of terms. PNLS may not have felt it was in need of “capacity building” in ways that USAID thought it was, and the scope and limits of BHAPP’s "technical assistance" were not clear. In part, the team felt it resulted from the lack of an effective bridge between BHAPP’s 3 Normes et Directives Nationales du Dépistage du VIH/SIDA et de la Prise en Charge Biologique des PVVIH, November 2004. BHAPP FINAL EVALUATION REPORT, JUNE 2006 7 support to the central PNLS and its activities in the field (described below.) This was reflected, for example, in comments by PNLS officials to the evaluation team to the effect that BHAPP technical resource people, despite their proximity, were not as consistently available to them as they felt they should be. (“Ils doivent être plus disponible.”) It was the team's conclusion that this was because the BHAPP technical people were often in the field, and inadequate attention was given, on both sides, to sharing of work schedules and coordination of activities. In this regard, the role of CIPECs as the interface between policies and priorities of central PNLS and field experiences rolled out under BHAPP was, in the team's view, not well explored or exploited. How might this have been different? As their name (Centre d'Information, de Prospective et de Conseil) suggests, CIPECs serve as regional centers (there are five in all) that transmit information and guidance for HIV prevention from the central PNLS to health facilities and, more particularly, health workers in their regions. Presumably they oversee health worker performance in using this information and guidance. CIPECs also serve as model regional service centers for management of STIs and voluntary counseling and testing (VCT) for HIV. This role would naturally seem to include overseeing dissemination of norms and standards developed by PNLS and BHAPP for management of STIs, referral for VCT and promotion of BCC. Because BHAPP field initiatives were extended only to selected zones sanitaires and, within those zones, to selected health facilities, CIPECs would not necessarily have been expected to play a supervisory role. On the other hand, it is logical to think that CIPECs should, as PNLS' decentralized representatives, monitor the progress of BHAPP interventions so as to learn all they could about them, and transmit their assessment of the efficacy of these interventions to PNLS. They should also be testing the new norms and standards in the services of their own model clinics. CIPECs should be regularly briefed by BHAPP on its activities, be invited to participate in supervisory visits, and otherwise do everything possible to maximize capacity building throughout the system. Lesson: It is important to have clear agreement between parties on the definition and scope of key operational terms, such as “capacity building” and “technical assistance”, and for everyone to work assiduously to coordinate activities between the parties. Lesson: A more effective bridge between learning from technical innovations in the field and strengthening PNLS at the center can be achieved by maximizing the oversight and analytical role of CIPECs as decentralized arms of PNLS. Lesson: BHAPP's work with PNLS has created a body of norms, standards and other materials of high quality, that are ready for use and wide dissemination as part of the nationwide HIV/AIDS program as well as under a new USAID-funded HIV prevention effort. 2. Enhancing the Policy Environment At the central level BHAPP also contributed substantially to improvement of the policy environment within which the national HIV and AIDS prevention effort must function. As described in the original project document, this involved a variety of interventions intended to (1) enhance national policy dialogue regarding HIV and AIDS programs; (2) strengthen partnerships and collaboration between national institutions and donor agencies; (3) encourage advocacy initiatives; and (4) “use information generated by a second-generation surveillance system to inform … intervention strategies.” In this regard, the project's numerous contributions included: • mapping HIV/AIDS-related activities of all funders in Bénin, to enhance PNLS' ability to coordinate their interventions and avoid duplication; • testing surveillance systems as an integral part of BHAPP fieldwork, ensuring dissemination of BSS I, and providing BHAPP FINAL EVALUATION REPORT, JUNE 2006 8 technical, logistical and editorial support to preparation and dissemination of BSS II; • coordinating and funding, in collaboration with other donors such as UNFPA, the finalization of Law 2005-31, the national law on prevention, management and control of HIV/AIDS in Bénin (this contribution was much praised by the First Vice-president of the National Assembly in an interview for the BHAPP newsletter); • providing logistical and technical support for revising the national HIV/AIDS Cadre Stratégique. By their very nature, many of BHAPP's activities on issues and tasks affecting the national policy environment supported the multi-sectoral, policy-related role of the CNLS as much as it did the public health-focused role of the PNLS. Given that BHAPP was designed to support and work with the latter (as the CNLS did not exist at the time the project was launched), this caused some confusion and had the potential for duplication of effort, although it seems that this was held to a minimum. What it did show is that, in future, differentiation between the roles and support needs of PNLS vis à vis CNLS must be made clear as regards inputs of an externally funded source. Lesson: The nature of supporting links to PNLS and CNLS must be differentiated and clarified to avoid conflict. Externally funded assistance to PNLS should be designed to be health-related and technical in nature, and support to CNLS designed to be multisectoral and strategic in nature. III.B. PROJECT DESIGN AND IMPACT – REGIONAL AND COMMUNITY LEVEL The evaluation team spent considerable time meeting and talking with health workers, NGOs, elected officials, peer educators and members of communities in zones sanitaires, or health zones, targeted by BHAPP. It also reviewed service statistics and surveillance data gathered by the project. From all of this the team concluded that, through interventions described in this section, BHAPP made a significant contribution to expanding awareness and understanding of STIs, HIV and AIDS, and to improving quality of prevention and treatment services, in its intervention sites. It also concluded that the shortness of the project as a whole, and its decision to work in only selected communities across multiple zones sanitaires, limited its potential for broader, more sustained impact. 1. Management of Sexually Transmitted Infections (STI) Technical and follow-up assistance provided by BHAPP for the prise en charge of STIs was greatly appreciated by the health workers it reached, and reflected itself in improved service statistics from all 23 cliniques or health facilities selected for intervention in targeted health zones. At the outset of the project, an evaluation of these facilities by BHAPP determined that a majority of health workers (doctors, nurses, midwives) were not trained in syndromic management of STIs, and performance of those that were trained was inadequate. Inputs designed by the project to correct these weaknesses included: • development of training protocols and training of 45 clinical trainers (TOT) • training of health workers in syndromic management of STIs, infection prevention and counseling (160 health workers trained); • preparation of detailed, technically up-to￾date national algorithms for syndromic management of STIs; • equipping of health facilities with algorithm Aides Memoires, wall posters, flip charts, films and other informational and technical materials needed to enhance STI services and counseling. An additional, much praised innovation was the introduction and implementation of the concept of supervision facilitante (supportive supervision). At least 63 health workers were trained in this BHAPP FINAL EVALUATION REPORT, JUNE 2006 9 approach to supervision, one that emphasizes regular follow-up and continuous dialogue as the best methods of assisting health workers to master the elements of service delivery of acceptable quality. These include STI management algorithms, infection prevention practices, patient education and counseling, and drugs management. Health workers contacted by the team agreed that supportive supervision, as introduced by BHAPP, had demonstrably strengthened their capacity to deliver comprehensive, high quality services, a fact confirmed by statistics. (See Annex 2, which summarizes project results against targets and indicators.) Surveillance of BHAPP-supported health facilities reflected steady improvement on the part of staff in STI case management, infection prevention, and, importantly, communication of HIV prevention messages and referrals for HIV testing.4 If health facilities could be equipped to also provide counseling in prevention of mother-to-child transmission of HIV (PMTCT), and to actually offer voluntary HIV counseling and testing (VCT), their impact would be even greater. Clinics must also continually grapple with the problem of drug supply. Since BHAPP, as a USAID-funded project, was not allowed to provide the drugs required for treatment of STIs, health facilities had to depend on a public sector supply system that is frequently behind schedule, and all reported stock-outs. Annex 2 reflects both the improvement in service quality, and the fact that no health facilities were without ruptures de stock (stock-outs) during the life of the project. The fact that BHAPP's STI support activities, including training, equipment and supervision, was limited to only 23 health centers within certain zones sanitaires in the five departments covered by the project caused resentment among unsupported health centers. In future, it would seem to make sense to opt for more 4 Graphs and charts reflecting evolution of performance in STI case management of health facilities supported by BHAPP are available from BHAPP reports, c/o Africare. comprehensive coverage in a more limited area. At the same time, tools and protocols that have been fully vetted and are of high technical quality, such as the Aide Memoire that sets out STI treatment algorithms, should be made available to services nationwide through the intermediary of CIPECs. Lesson: It is preferable to provide full STI (or other) service coverage within a more limited number of zones sanitaires, to maximize impact and avoid resentments. However, STI standards developed, refined and implemented under BHAPP should be applied nationally by PNLS/ CIPECs as much as possible. Lesson: If STI case management can be supplemented by VCT and PMTCT counseling and services, and even linked to reproductive health services, it can play an even more effective role in preventing HIV transmission. 2. NGO Strengthening for Behavior Change Communication (BCC) The peer educator model. Through partnerships established with indigenous NGOs in the five target departments, BHAPP implemented a model of behavior change communication that centered on training peer educators to take messages of behavior change for preventing HIV transmission to target groups in the community deemed to be at high risk of infection. To make this possible, BHAPP provided NGO managers and outreach coordinators (animateurs) with training, materials and modest financial support through a series of “micro-projects.” Each micro-project was four months in length, followed by a period to evaluate its impact before the next was launched, with a different target group. Target groups from which peer educators were recruited included: • hair dressers (coiffeuses) • truck drivers • bicycle taxi drivers • employees of hotels de passage frequented by sex workers BHAPP FINAL EVALUATION REPORT, JUNE 2006 10 • artisans' associations The peer educators were given two days of training in the specifics of HIV transmission and its prevention, and their travel expenses and per diems were covered. Based on the BCC norms and standards developed by BHAPP with PNLS (described above), peer educators were trained in different approaches to communication of relevant messages, and in how and where to refer individuals who, based on these messages, decide to seek STI treatment and/or HIV testing. Peer educator trainees were given flip charts, films and other informational materials for use in transmitting messages. They also received a starter supply of condoms, which they were permitted to sell, using part of the profit for re￾supply. They then returned to their communities, under supervision of NGO outreach coordinators, to hold causeries, or group discussions, and conduct other group and one-to-one outreach to transmit the message of HIV prevention to target groups. The process was well thought-out and, by all indications, effective. The one gap that the team observed, which might have been filled were the project to have lasted longer, was in helping outreach workers learn to tailor messages to respond to the particular characteristics of the local culture and community. Each community and group of people has its own personality, and a standardized approach to message creation does not always acknowledge this reality. The evaluation team was struck by the purity of this NGO-based, peer educator-centered BCC model. It is simple and low cost and thus more sustainable than other models that include financial allowances, t-shirts, and/or other inducements for outreach workers. This is not to say that such inducements are not welcome. NGOs frequently made mention of the modest level of financial investment provided by BHAPP in comparison with other BCC initiatives, such as the PPLS program of the World Bank. At the same time they admitted that the BHAPP model is more sustainable over time. They were appreciative of the quality of BHAPP’s oversight, which, project surveillance showed, strengthened the management capacity of all 25 partner NGOs. And they were proud of having created a cadre of knowledgeable, motivated peer educators - over 1,500 trained and fielded under the project - as a resource now permanently in place. 5 A weakness of this component of BHAPP was the phased “micro-project” model for NGO BCC activities, featuring four-month periods of activity interspersed with three to four weeks for review and evaluation. The evaluation team found this to be overly controlled and rigid, limiting target group selection and preventing smooth expansion of activities as capacity of implementers was strengthened. One NGO director put it very succinctly in his conversation with the team: “The approach of BHAPP is very good, in that it permits direct contact with the people. But (at the same time) it is too limited. There are so many groups that we could not contact (because of the limitations of the micro-project structure.) Ambulatory sales people, young women, school youth … pourquoi pas toucher tout le monde?” NGO selection. As to selection of partner NGOs, at the outset BHAPP selected six relatively mature NGOs for training, with the intention that they would in turn establish mentoring relationships with 20 local NGOs to implement BCC activities. But this approach proved cumbersome, and the differentiation between NGOs confusing. BHAPP shifted to the simpler approach (fortuitously, in the team's view) of building partnerships directly with about 25 NGOs through training, supportive supervision and funding of micro-projects. Non-governmental organizations selected as BHAPP partners were, on average, 10 -15 years old and had been BHAPP grantees for two to three years. Most NGOs have been established 5 As an expression of its appreciation for the work of the peer educators, BHAPP did decide to reward each of them with a specially monogrammed outreach worker “kit” filled with information materials and supplies, at the conclusion of the project, which the evaluation team found a very generous gesture. BHAPP FINAL EVALUATION REPORT, JUNE 2006 11 around a mission of community-centered social and economic development, as indicated by the names of a sample: CEPADES Center for Reflection and Action for Economic and Social Development (Commune d'Allada, Département d'Atlantique) CBFORD Bénin Center for the Environment, Training and Local Development (Zogbodomé, Zou) GRADEL Group for Research and Support to Local Development (Glazoué, Collines) GRADID Group for Research and Action for Lasting and Integrated Development (Dogbo, Couffo) Typically, NGOs have a staff of three to five persons (nominally paid, if at all), and are almost wholly dependent for functioning on external donor assistance, which some are more successful than others in attracting. Generally NGO leaders have established good working relationships with elected officials and traditional leaders in their communities, which puts them in a good position to tackle difficult issues such as prevention of HIV. NGO managers expressed great appreciation for the quality and consistency of BHAPP support. Said one, “We have all learned something … that must be recognized.” At the same time, many expressed the feeling that their financial support “package” should have been more generous, offering many suggestions as to what it might have included. But again, the team felt otherwise. The more that external support is kept to an essential minimum, the greater are the chances that the work it supports can be sustained into the future. In general, the team felt that NGOs are an important focal point for field activities, but their capacities and skills are inevitably varied. Other community focal points should be used to broaden outreach, such as traditional chiefs, elected officials and traditional healers. BHAPP did this to an extent, but could have done more. Lesson: The simpler, more focused and less costly an outreach model can be, the better its chances for sustainability. The relatively "pure" peer educator model implemented under BHAPP is worthy of replication. Lesson: It is important to develop a BCC outreach and implementation model that can facilitate unbroken growth and expansion within all appropriate target groups, one that is not hindered by adhering to artificially short sub￾grant units. It is also important to give outreach workers the skills and understanding needed to tailor, and continually revise, their messages according to the particular characteristics of their local culture. Lesson: NGOs are one logical and appropriate focal point for implementing HIV prevention activities in the field, but other community leaders and entities, such as traditional chiefs, elected officials and traditional healers, should also be approached and involved. 3. Enhancing Social Marketing The BHAPP CA called for the project to “coordinate with PSI’s contraceptive social marketing (CSM) project to facilitate condom distribution by community health workers and peer educators.”6 On completion of training, peer educators were given starter supplies of condoms by their NGOs, which had been provided with a budget for the purpose as part of their support from BHAPP. Peer educators were urged to sell condoms within target groups with which they worked, and thereby create a revolving fund that would give them some income while enabling them to purchase new supplies. It was also intended that they would be oriented to local social marketing outlets, where low-cost condoms were available, so that they could refer their contacts to them in the future. All sides agree that this aspect of outreach by BHAPP partner NGOs was inadequately 6 From BHAPP Cooperative Agreement, p. B-5. BHAPP FINAL EVALUATION REPORT, JUNE 2006 12 developed. From what the evaluation team learned, it was clear that peer educators were not well trained in how to establish condom sales and build up a revolving fund. When asked, they were also not able to identify social marketing outlets in the community. Typically, they had heard about PSI but were not well versed in the CSM concept. Ultimately, the amount of condoms distributed (approximately 50,000) was modest and the process not sustained. Lesson: CSM is an important element for sustaining BCC impact. In the future, collaboration with the national social marketing effort needs to be substantially enhanced, in terms of training of peer educators, orientation to concept of CSM, and identifying and strengthening links between NGO outreach workers and vendors. III.C. OTHER IMPLICATIONS / RECOMMENDATIONS FOR USAID-FUNDED ACTIVITIES Giving More Than Lip Service to the Issue of Gender The issue of gender imbalance deserves less lip service and more vigorous, concentrated attention in projects of this nature. For example, it was impossible for the evaluation team to ignore the fact that none of the many NGOs with which it met had any women “at the table,” i.e., in management positions of any importance, and that none of the leadership of health facilities it visited was female. Even among peer educators, women were in a distinct minority. Women bring unique life skills to projects such as BHAPP, skills that are essential to effectively reaching a major portion of the target audience. Although the idea of quotas may strike some as inappropriate, it is equally clear that, if there is to be any change (in this or projects like it), the issue of gender imbalance must be addressed with firmness and specificity. It is urged that NGOs and others implementing BCC activities be required to train equal numbers of female and male peer educators (if not a preponderance of females). NGOs should also be required to have at least one woman, preferably more, in senior program roles (e.g., director, adjoint, administrator, and/or animateur). Integrating Services Linked to BHAPP was a collaboration between Africare and Médecins du Monde (MDM) known as the Projet Intégrée de VIH Ahéme (PIVA.) Implemented in the zones sanitaires of Comé and Ouidah, PIVA focused not only on STI case management and HIV prevention, but also offered services for PMTCT, VCT, and care and support for people living with AIDS and their families. The high degree of appreciation for PIVA in the health zones where it operates attests to the value inherent in offering as broad and integrated a range of services as possible, rather than adhering to a narrow focus on STIs. Gaps in Coverage The evaluation team quickly became aware of a large impending gap in external support for HIV programming in Bénin. Not only has BHAPP ended, but the Canadian-funded SIDA-3 project and assistance for HIV prevention provided by Coopération Française will shortly be terminated as well. One key target group that will, as a result, not be well covered is sex workers, which have the highest rate of infection among all high-risk groups in the country. Sex workers were the major focus of SIDA-3 and benefited as well from the other programs. As USAID develops its new HIV prevention and support program, and determines its geographic and technical focus, it would seem that this will be an important group to include. BHAPP FINAL EVALUATION REPORT, JUNE 2006 13 IV. IMPACT OF BHAPP VS. PERFORMANCE MONITORING PLAN (PMP) The evaluation team found the BHAPP PMP, and the indicators against which progress was measured, to have been generally helpful in gauging project impact. Annex 2 summarizes information on results of BHAPP interventions, based on data gathered through continuous internal evaluations by BHAPP of its micro￾projects and clinical interventions, as well as more extensive surveillance activities. Foremost among the latter was a 2005 KAP study spear￾headed by BHAPP, in collaboration with PNLS, in zones sanitaires covered by the project. 7 Once again, the principal weakness in all of this was that surveillance was only on selected target sites within broader zones and departments, rather than on a concentrated sample. However, it did show what could be accomplished, in terms of measurable, and sometimes significant, results. Some highlights: In terms of increased knowledge of prevention measures and appropriate behaviors, a substantial increase was recorded among truck drivers in condom usage between 2002 and 2005, along with an equally impressive gain in drivers’ ability to name at least two methods of reducing risk of HIV transmission. On the other hand, condom usage among adolescents remained static, and that among sex workers declined. This would seem to indicate the need for greater creativity and flexibility in tailoring messages to different groups of adolescents, sex workers and the like. As mentioned earlier, this was the principal weakness that the evaluation team perceived in the project’s BCC outreach - the need for more training, especially of peer educators, in constantly adjusting their presentations to meet changing cultural and other characteristics of target audiences. 7 Evaluation des Connaissances et Pratiques du VIH/SIDA/IST dans les Zones Couvertes par le BHAPP, produced through the Bureau d'Appui en Santé Publique '96, December 2005. Progress in health worker performance was perhaps the most striking success of the project. The percentage of health workers capable of properly diagnosing and treating STIs, and counseling those infected with STIs and/or HIV, increased from about 4% in 2002 to 33% in 2004 and 40% in 2005. Percentage of health workers able to properly explain STI prevention and treatment norms grew from 15% in 2003 to 47% in 2004. The figures remain well below what must be the ultimate goal of 100%. But, along with anecdotal information gathered by the team, they indicate that implementation of the STI component of BHAPP interventions was well conceived and led to a major increase in service quality. Unfortunately the figures also reveal a significant weakness in the supply and commodity distribution system, one that adversely impacts service quality at health facilities providing STI diagnosis and treatment. Throughout the life of the project, no health centers were spared shortages in stocks of STI drugs, a problem linked to weaknesses in national distribution systems. This is an issue over which neither the facilities nor BHAPP had control, and it would seem to fall to PNLS to do everything possible to avoid this problem in future. Finally, improvement in the policy environment was reflected in completion of several tasks. These included: dissemination of BSS I and preparation, revision and publication of BSS II; development of a range of detailed norms and standards for various aspects of HIV prevention and STI service delivery; and assurance that as of 2003 the PNLS was operating according to a detailed operational plan. Lesson: The BHAPP PMP provided a realistic, manageable number of indicators by which its impact could be measured. With the exception of the issue of drug supply, it established targets that challenged the project, and provided a useful guideline for future project design. BHAPP FINAL EVALUATION REPORT, JUNE 2006 14 V. COLLABORATION WITH COUNTERPARTS AND STAKEHOLDERS The evaluation team was struck by the frequency with which BHAPP, and in particular its Chief of Party, was lauded by other major stakeholders, national and international, for the central role it played in keeping everyone informed and on track in the campaign against HIV and AIDS. This was above and beyond BHAPP’s planned, and largely successful, efforts to enhance the policy environment for HIV and AIDS programs, as discussed above in section III.A. Filling this collaborative role meant taking initiative to convene meetings when needed to deal with key issues, providing funding or logistic support to complete a publication or other activity, and/or otherwise helping to move things along. UNAIDS, the World Bank, and the bilateral donors with whom we talked all spoke of the key role BHAPP had played in coordinating everyone's efforts. In the process it helped create an unusually positive, collegial level of interaction among all of the donor agencies active in Bénin, and a notable absence of the interagency bickering which, in other circumstances, can be very debilitating. Lesson: Consistent, harmonious interagency collaboration can have an enormously positive effect on the quality of a national program, and on relations with the host government. This is a reality that cannot be overstated. BHAPP FINAL EVALUATION REPORT, JUNE 2006 15 VI. PROJECT MANAGMENT General. Given that the project was about to be closed (and now has been), the evaluation team spent little time looking at the day-to-day management of BHAPP, since recommendations there would have had no time to take effect. However, as stated in the Acknowledgements section of this report, we were impressed with the pride of the BHAPP staff in what they had accomplished, and with their willingness to share it with us. To us this highlighted an organizational culture that was creative, forward looking and focused on results, and one can only hope that any follow-on project will benefit from a similar level of motivation. While the project's Chief of Party was changed midway through its four-year life span, this does not seem to have caused any major disruption. Indeed the final half of the project seems to have been as productive as the first, if not more so, and BHAPP’s very positive role in promoting constructive collaboration within the donor community, as described above, became ever more significant. The team expressed its views earlier in this report with respect to the “micro-project” structure of BHAPP’s support to NGOs and their BCC outreach activities to selected target groups. This seems to have been an unduly restrictive approach to managing sub-grants, one that limited flexibility and scope, and that we would urge not be repeated. Finances. A significant financial setback to the project came in its last, phaseout year of operation, when USAID/Bénin reduced its final incremental funding for the cooperative agreement from $700,000 to $300,000. As the evaluation team understood it, this stemmed in large measure from a crucial delay in financial reporting on the part of Africare’s Washington office, which led USAID to understand that the project's needs were less than had been estimated and agreed. When that proved not to be true, USAID had already reprogrammed those funds and they could not be reinstated. Africare/Bénin ended up covering as much of the shortfall as it could from other resources, and some intended phaseout activities had to be curtailed. Project reporting. BHAPP project implemen￾tation included production of an enormous number of reports and other documents, far more than would seem to have been required under USAID/Bénin's reporting requirements. In the future, the burden of project reporting and documentation should be carefully analyzed and kept to a minimum, in actual numbers of reports as well as in page length. Staff members could then be freed to spend more time in the field supervising, analyzing and encouraging health and outreach workers. Mid-term evaluation. The evaluation team reviewed the mid-term evaluation of BHAPP conducted in July 2004, and interviewed a member of that evaluation team. One of the key recommendations of that team was for the project to reduce the number of departments in which it was active from five to two, to make possible a greater concentration of resources on a more limited target. The recommendation was not acted upon, but it coincides with our own observation that the project's selection of target zones sanitaires was too spread out and fragmented, limiting impact and causing resentments. Two other key suggestions were acted upon, with largely positive results. As noted, two of BHAPP's technical experts were eventually housed at PNLS, a strong recommendation of the mid-term evaluation. And the recommen￾dation that the project publish a regular bulletin or newsletter was also taken up with the publication of two issues of BHAPP in Action. The latter was a positive, creative addition, and should be a part of any future project. Lesson: Inattention to financial reporting requirements can be very damaging to a project. Lesson: Project reporting requirements, whether real or self-imposed, should always be carefully analyzed and kept to an appropriate minimum, to ensure that project personnel are able to spend as much time as possible launching, supervising and assessing field activities. BHAPP FINAL EVALUATION REPORT, JUNE 2006 16 VII. CONCLUSION The BHAPP experience has taught many lessons, detailed in the preceding pages, that are political, strategic, and/or technical in nature, and in each of which are embedded recommendations for the future. These include, but are by no means limited to: • defining terms (e.g., "technical assistance" and "capacity building") in the interest of harmonious and productive working relationships; • defining roles and linkages (e.g., between PNLS and CNLS), to avoid duplication and confusion; • making the most of the PNLS structure so that decentralized services and innovations inform central decision-making; • rethinking the structure of sub-grants to NGOs, as well as broadening the base of community groups through which support is provided; • understanding the importance of local community- and culture-specific design of BCC messages, and reflecting that in training; • not fragmenting clinical service inputs, but rather choosing target areas so as to ensure full coverage of high quality STI services, and hopefully widening the range of services; • being sure that the excellent technical work that has been done, and the norms, standards, algorithms, training curricula and other technical materials and resources that have resulted, are used to the fullest, rather than cast aside for something new. A final word is in order about the core mission of BHAPP, namely to reinforce the capacity of the PNLS "to coordinate and manage a nationwide HIV/AIDS program." Despite the fact that the working relationship between PNLS and BHAPP could have been closer and more collaborative, nonetheless the genuine appreciation expressed by PNLS for BHAPP, and the evidence of productive results from this partnership, indicated to the evaluation team that this is the way to go. External support must be geared to building host country capacity, and when it comes to leading the campaign to strengthen health services to combat HIV and AIDS and better manage STIs, PNLS is the only game in town. The CNLS does have a key role to play, in an overarching, multi-sectoral sense, in expanding awareness, building political will, and overseeing the strategic development process. But the PNLS must lead where it matters most - at the level of the community and the health center, where people make decisions about their health-related behaviors and health workers must treat and counsel them. Given that reality, external assistance must be geared accordingly, starting with a full sharing with PNLS of ideas and competencies, and followed by the hammering out of a working relationship that makes the absolute most of the resources available. BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEXES ANNEXES I. PERSONS MET AND SITES VISITED 2. BHAPP RESULTS vs. TARGETS AND INDICATORS 3. EVALUATION SCOPE OF WORK BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX I ANNEX 1 PERSONS MET AND SITES VISITED Département du Mono/Couffo OMEGA ONG (Commune de Comè) : Aïssan Raymond............................................................Directeur Exécutif Garba Séïdou.................................................................Animateur Mathe Mensah ...............................................................Comptable ARED ONG (Arrondissement de Sê) : Paul Kounoudji ..............................................................Directeur Exécutif Housounou Flavien.......................................................Animateur Mathieu Yénou..............................................................Animateur CSA de Sê : Charlemagne Odjo.......................................................Infirmier major CIPEC Mono-Couffo (commune de Lokossa) : Dr Hugues Guidibi ......................................................Responsable du CIPEC ARRONDISSEMENT de Dogbo Tota : Sêgnannou H. Emmanuel.............................................Chef d’arrondissement GRADID ONG (Commune de Dogbo) : Albert Edou....................................................................Directeur Exécutif Pauline Gnassounou.....................................................Animatrice DDSP Mono/Couffo : Dr Charles Sossa .........................................................Directeur Départemental de la Santé Publique PAIR EDUCATRICE (Arrondisement de Sê) : Virginie Sossou..............................................................Maîtresse-coiffeuse Département du Zou-Collines GRADEL ONG (Glazoué) : Emile Fassoundé............................................................Directeur Exécutif MAIRIE de GLAZOUÉ : Djim Atchikpa................................................................Maire de Glazoué CSC de Glazoué (Site IST) : Dr Séraphin Ahoui .......................................................Médecin-chef CIPEC Zou-Collines : Dr Joachim Aïfa.............................................................Responsable du CIPEC CECO ONG (Savè) : Kélani Balogoun ............................................................Directeur Exécutif Jean-marie Boni.............................................................Animateur Alassane Mama..............................................................Animateur ALDIPE ONG (Bohicon) : Jules Béhanzin................................................................Coordonnateur Euloge Chaffa.................................................................Animateur Placide Godovo.............................................................Animateur CBEFORD ONG (Abomey) : Sébastion Aïmihouè......................................................Animateur Bernardin Tchibozo .....................................................Animateur CSC de ZOGBODOMEY(Site IST) : Dr Bruno Aholoukpê...................................................Médecin-chef Département du l’Atlantique EEZS d’ALLADA (Site IST) : Hyacinthe Amédomè...................................................Méd-coord de la ZS d’Allada Honorine Zoblikpo ......................................................Prestataire/superviseur Damienne Donoumassou ...........................................Prestataire/superviseur BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX I CPADES ONG (Commune d’Allada) : François xavier Avaho .................................................Animateur Aholidji Agossou...........................................................Animateur OPESVAT ONG (Commune d’Allada) : Félicien Hakpon.............................................................Animateur Albert Zoblikpo ............................................................Directeur Exécutif LAMA PROGRÈS ONG (Arrondissement Sêhouè) : Gaston Amoussouga....................................................C/P Lama-progrès Edwige Ahouansè..........................................................Animatrice PAIR EDUCATRICE : Adandjèkpo Collette....................................................Apprentie coiffeuse HOPITAL DE ZONE (Ouidah) : Gratien Aguessy............................................................Directeur de l’Hôpital CERPADEC ONG (Commune de Ouidah) : Albert Massenon...........................................................Directeur Exécutif Marcel Amoussou.........................................................Animateur PAIRS EDUCATEURS/TRICES : Marie Kouyè ..................................................................Pair éducatrice Tatiana Lègba.................................................................Pair éducatrice Armand Yénonfan.........................................................Pair éducateur Noé Codo......................................................................Pair éducateur Organismes et Personnes Rencontrées PNLS Dr Marcel Zannou ...................................................Coordonnateur Dr Edgar Lafia............................................................Coordonnateur Adjoint Marie Constance Mélomè ......................................Chargée CCC Guy Laruche (Coopération française).................Assistant Technique CNLS Dr. Mèdégan Kiki Valentine ..................................Secrétaire Permanent Jonathan Amégnigan.................................................Responsable Cellule Santé SIDA 3 Dr. Marguerite Ndour ............................................Coordinatrice Nationale PPLS Dr. Olivier Capo-Chichi .........................................Coordonnateur National PSI Jim Malster.................................................................Country Representative AFRICARE Abdallah Meftuh........................................................Country Representative ONUSIDA Dr Yamina Chakkar.................................................Country Representative BHAPP Mbella Ngongi ...........................................................Chief of Party Edmond Kifouly ........................................................CCC Specialist Dr Séraphin Vissoh ..................................................Deputy Chief of Party Dr Karim Seck ..........................................................IST Specialist Odile Sodoloufo .......................................................NGO Specialist….(et al) USAID Donald Dickerson....................................................Deputy, USAID FHT Pascal Zinzindohoué ................................................USAID FHT Leader Charles Ogounchi ....................................................USAID FHT AUTRES PERSONNES RESSOURCES .....................................................Dr Emile Akowanou, BHAPP mid-term evaluator BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 2 ANNEX 2 BHAPP RESULTS vs. TARGETS AND INDICATORS N° Objectives Strategies Results Target Groups Indicators Achievements 1. Revised (second generation ESDGB) surveillance system in place Truck drivers, in-school and out-of-school youth, sex workers 1. ESDGB available and disseminated 1.1 BSS1 report available and disseminated 1.2 ESDGB report available and dissemination in progress 2. Government staff and other decision makers use viable data on STIs and HIV PNLS, SP/CNLS and decision makers 2. Number of changes made in policy, standards and programs due to BSS 2.1 Standards and procedures document (05) 2.2 Definition of 19 national indicators by SP/CNLS 1 Improve the Political Environment • Plan • Partnerships • Technical Assistance 3. Strengthen the capacity of PNLS to coordinate the national fight against AIDS PNLS 3. PNLS has a workplan detailing involvement of different partners Since 2003 PNLS has and operational health sector plan (Plan Opérationnel Secteur Santé - POSS) 1. Improvement of supply and distribution system 2. Integration and expansion of family health services in health facilities 3. Expansion of services and distribution of commodities at the community level Out-of-school youth and dropouts, mobile populations, sex workers Percentage of health workers who are capable of correctly explaining prevention standards and caring for STI/HIV cases 2003: 15% 2004: 47% 2005: Percentage of targeted health centers that have had no stockouts of STI medications in the last 12 months 2003: 0% 2004: 0% 2005: 0% 2 Increase Access to Services and Commodities • Training • Monitoring and Evaluation • Needs Assessment Percentage of high performing peer educators active in BCC activities 2005: 91% to 99% BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 2 N° Objectives Strategies Results Target Groups Indicators Achievements 3 Strengthen service management quality. • Dissemination of standards • Training • Monitoring and evaluation • Needs analysis Management capacity increased. Better performance of health care providers Health service staff covered by BHAPP Percentage of health care providers who are capable of correctly treating STI/HIV clients and counseling them on caring for STIs and HIV Percentage of targeted health centers that had no stockouts of STI medications in the last 12 months 2003: 4.7% 2004: 32.8% 2005: 40% 2003: 0% 2004: 0% 2005: 0% 4 Increase demand and support for usage of services, commodities and preventive measures • Technical Assistance • Training peer educators • Monitoring and evaluation • Grants Improve the knowledge and practices of target groups. Out-of-school youth and dropouts, mobile populations, sex workers Percentage of individuals in targeted groups who say they have had at least one sexual partner outside of their regular sexual partner in the last 12 months. Percentage of individuals in targeted groups that say they have used a condom during their last sexual encounter with an occasional partner in the last 12 months. Percentage of individuals in targeted groups that are capable of listing at least two risk reduction methods for sexual transmission of HIV 2002 2005 Truck Drivers 35% 43.3% Adolescents 6-20 % 20-50% 2002 2005 Truck Drivers 14-39% 68.2% Adolescents 35-49% 37-42% Sex Workers 90% 82.8% 2002 2005 Truck Drivers 7% 40% Adolescents 4.8-9% 9-10.3% BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 3 ANNEX 3 EVALUATION SCOPE OF WORK A.1 BACKGROUND A.1.1 Current USAID/Benin Health Portfolio: Health statistics in Benin, which has a population estimated at 6,769,914 inhabitants (cf 2002 census), reveal a worrying situation. For example, child malnutrition is high, with 29% of children under 3 underweight, and treatable illnesses, such as malaria, diarrhea, and acute respiratory infections, are the major causes of infant morbidity and mortality. Fertility rates (5.6 DHS 2001) and population growth (2.5% WB 2004) are still high while contraceptive prevalence is low (7.2% DHS 2001). Although the HIV prevalence rate is relatively low (2.0%; cf. PNLS-2004) compared to other countries in the sub-region, it could rise rapidly without a coordinated and appropriate response. Segments of the population remain inadequately served by the public health system. While public health service coverage remains low, the private health sector, both profit and non-profit, has grown rapidly, especially in urban areas. The premise of USAID/Benin’s health program is that improving access to, quality of, and demand for health services, will lead to their increased use. These goals are achieved in an enabling policy environment that allows for the decentralized management of sustainable family health services. Thus to achieve USAID/Benin’s health Strategic Objective (SO), the program promotes and reinforces policies, norms and standards at the national level, which facilitate the implementation of an integrated program of activities at the intermediate (departmental) and peripheral (health zone) levels. USAID/Benin’s health program for the previous strategy period focused on the delivery of integrated family health services to targeted populations. The Benin Integrated Family Health Program was developed in close collaboration with the MOPH and included four primary bilateral projects: the Regional Integrated Family Health Program (PROSAF); the nation-wide Integrated Social Marketing Program (PRIMS), the Benin HIV/AIDS Prevention Program (BHAPP) and the Africa Integrated Malaria Initiative (AIMI). These bilateral activities were complemented by field support initiatives focusing on improving maternal health through safe delivery techniques (ACQUIRE Project), and decentralization and community financing of health care (PHR+ Project). Moving forward, USAID/Benin will build on the successes and strengths of the Benin Integrated Family Health Program. In order to rationalize resources and streamline program management, USAID is consolidating its health program and the majority of activities will be implemented through two major bilateral agreements, while calling on centrally-funded mechanisms for assistance as needed. The first of these bilateral programs, the Integrated Family Health Program, will cover the delivery of health services through the public sector. It will focus on expanding the minimum package of health care services provided at health facilities, also including postpartum hemorrhage and emergency obstetrical and neonatal care; scaling up best practices in malaria prevention and treatment; integrating HIV/AIDS with other health services; scaling up and improving integrated management of childhood illness (IMCI) services; working to ensure RH commodity security; strengthening health mutuelles; and supporting decentralization of the MOPH. These program components are described more fully in section IV, B (on complementarity) below. BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 3 The second major bilateral program the mission will support, the Integrated Social Marketing and HIV/AIDS Program, will focus on product marketing, service delivery and communication through the non-governmental and private sectors. The HIV/AIDS component will include the development and implementation of two pilot sites that will provide “best practices” for integration of HIV/AIDS, STI and FP services. The social marketing component will promote and sell subsidized commodities for FP and prevention of malaria, diarrhea, STIs, and HIV/AIDS, and expand a pilot franchised network of private health providers. The integrated program will also encourage healthy behaviors through mass media and interpersonal communication. The program will include a technical assistance component focusing on strengthening the capacity of the National AIDS Control Program (PNLS) and the National AIDS Control Committee (CNLS) to plan, implement, monitor, coordinate, and evaluate the national HIV/AIDS program, while advocating for policies that create a supportive environment for social marketing and the provision HIV/AIDS products and services. A.1.2 HIV/AIDS in the Benin Health Portfolio Since 2002, USAID has supported the Benin HIV/AIDS Prevention Program (BHAPP), (which ends in May 2006), which works at the national level to strengthen the management and coordination of PNLS and to promote HIV/AIDS prevention and STI case management in selected health zones. According to the cooperative agreement, the goal of the program is to reduce the rate of HIV/AIDS transmission in Benin. The objective of the program is to reinforce PNLS’ capacity to coordinate and manage a nationwide HIV/AIDS program. Activities designed to reinforce existing management and coordination mechanisms at the national and departmental levels include: advocacy for an enabling policy environment; BCC; training and technical assistance for STI case management; behavioral surveillance; and, in coordination with the Social Marketing CA, assistance for the social marketing and distribution of condoms by peer educators working with high risk members of the community. Among other achievements, BHAPP has been active at the central level in working with the PNLS to develop an annual workplan; the mapping of various partners’ interventions nationwide; and the development of norms and standards for BCC, STI, and other HIV/AIDS-related interventions. In partnership with PRIMS, SIDA3, and PNLS, BHAPP revised and disseminated BCC and educational materials. In addition, BHAPP supported second generation surveillance (serological surveillance combined with behavioral surveillance). At the decentralized level, six NGOs were trained to provide technical support to 21 other local NGOs to develop interpersonal BCC activities in local communities in the five departments of Atlantique, Mono/Couffo, and Zou/Collines. Among high risk groups, such as prostitutes, truckers, and motorcycle taxi drivers, 1,045 peer educators were trained, and as a result, approximately 30,000 beneficiaries were directly reached with BCC messages and 18,000 condoms distributed. In health care delivery, more than 3,500 STI cases were managed/treated in 23 BHAPP￾supported health centers where 160 health care providers were trained in the syndromic management of STIs. Over these last three years, the performance of the health care providers in counseling and support to patients and drug management improved dramatically. In June/July 2004, Africare carried out a mid-term evaluation which identified a number of constraints and challenges faced in the implementation of the project and made several recommendations for improving project performance. A.2 TITLE Final evaluation of USAID/Benin HIV/AIDS Prevention Program (BHAPP) BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 3 A.3 PURPOSE The purpose of this task order is to conduct a final evaluation the Family Health Team/USAID-Benin funded Benin HIV/AIDS Prevention and Care Project (BHAPP) implemented by Africare in collaboration with JHPIEGO. The findings of this evaluation will shape the implementation of any follow-on project. The project is scheduled to end in May 2006. A.4 STATEMENT OF WORK A.4.1 Terms of Reference: The contractor team shall conduct the evaluation, respond to all points included in this statement of work, and submit a final evaluation report. The contractor team must submit a report at the conclusion of the evaluation, which provides clear and concise findings, conclusions and recommendations. The evaluation report shall also provide a statement of lessons learned and future directions that may emerge from the evaluation. The Contractor team shall work closely with the Family Health Team (FHT), the members of BHAPP (both at Africare and JHPIEGO in the US and Benin), Government of Benin (GOB) counterparts (PNLS, MOHP, PNLS) and other donors active in HIV/AIDS in Benin, such as the French Cooperation and UNAIDS. The evaluation should determine whether the BHAPP achieved its goals and objectives including a review of whether the targets in the PMP were met or not. If goals, objectives, targets were not met, the evaluation should attempt to determine why this has happened. Specifically, the evaluation should look at: • Whether BHAPP is meeting the goals and objectives as set forth in the project, including whether the indicators in the PMP are appropriate and/or valid and whether the project has established reasonable methods for gathering data necessary to monitor progress and indicator data. • How project management affected the performance of the overall project. Describe strengths and weaknesses and identify contributing factors. • The effectiveness of collaboration with counterparts and other partner agencies. Describe strengths and weaknesses and identify contributing factors. • The appropriateness of the overall design of project interventions. Identify strengths and weaknesses. Identify aspects that should be carried over into a follow-on activity and explain why. The evaluation methodology will consist of: • Document review (see reference materials below) at USAID/Benin and BHAPP as well as with the GOB and other donors involved in HIV/AIDS. USAID/Benin TOCTO will provide all the reference materials as soon as requested; • Site visits to project-assisted STI clinics and NGO/peer education sites; BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 3 • In-depth interviews (and focus groups if deemed relevant and practicable) with stakeholders and beneficiaries. These include but are not limited to: o The other USAID-funded partner in HIV/AIDS prevention: Population Services International (PSI) o Participating NGO management and project staff o Participating clinic management and staff o Regional and zonal health management team members o Other major players in HIV/AIDS in Benin including: ƒ The National HIV/AIDS Committee (CNLS); ƒ The National AIDS Control Program (PNLS) including the CIPECs/“Centre d’Information, de Prevention, d’Education et de Conseil” (in Departments) ƒ UNAIDS ƒ UNFPA ƒ UNICEF ƒ The World Bank ƒ The European Union ƒ The French Cooperation ƒ The multi-sector HIV/AIDS Program funded by the World Bank (PPLS) ƒ The Corridor-Cross Border Project (Nigeria, Benin, Togo, Ghana and Cote d’Ivoire) ƒ The Project SIDA 3 funded by Canadian Cooperation ƒ Other agencies as appropriate o Benin’s civil society actors engaged in HIV care and prevention activities (although not necessarily directly participating in the BHAPP program), including but not limited to: ƒ Benin Health NGO Network (ROBS) ƒ Association ESPOIR et VIE (Association of People Living with HIV/AIDS) ƒ OSV-Jordan ƒ NGO Arc-en-ciel A.4.2 Reference materials: The following documents are recommended to the evaluation team as reading materials for the evaluation. The list is not exhaustive. • USAID/annual report Fiscal Year 2005 • The USAID/Benin HIV/AIDS Strategy BHAPP FINAL EVALUATION REPORT, JUNE 2006 ANNEX 3 • The Mission Current Country Strategic Plan (CSP) • The BHAPP mid-term evaluation report • The Demographic and Health Survey (DHS-2001) • The current Benin CSP • FHT’s current PMP • The Mission Gender Strategy • The BHAPP cooperative agreement • BHAPP quarterly and annual technical reports • The BHAPP PMP. • The National AIDS Control Program (PNLS) Strategic Plan • The two Reports of the Behavioral Surveillance surveys(CEFORP) • The National Epidemiologic Surveillance Reports/2001 – 2002 – 2003, 2004 (PNLS); • The “Report of the first HIV/AIDS/STI Surveillance of 2nd Generation on sex-workers and their clients in Benin” (SIDA 3). • Other relevant BHAPP documentation.