EVALUATION OF THE JAMAICA ADOLESCENT REPRODUCTIVE HEALTH ACTIVITY 2000–2007 DECEMBER 2007 This publication was produced for review by the United States Agency for International Development. It was prepared through the Global Health Technical Assistance Project. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Presented to USAID/Jamaica Judith Senderowitz Tijuana James-Traore Roxanne Johnson Karla Pearcy-Marston Jennifer Stuart-Dixon December 21, 2007 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 07-001-09) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources/. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 iii CONTENTS ACKNOWLEDGEMENTS ............................................................................................................................. v ACRONYMS ................................................................................................................................................ vii EXECUTIVE SUMMARY .............................................................................................................................. 1 I. INTRODUCTION ............................................................................................................... 5 Scope of Work .......................................................................................................................................... 5 Evaluation Team ....................................................................................................................................... 5 Methodology ............................................................................................................................................. 5 II. BACKGROUND ................................................................................................................. 7 III. USAID/J ASSISTANCE TO ARH SINCE 2000 ................................................................. 9 IV. FINDINGS ....................................................................................................................... 11 Youth.now ............................................................................................................................................... 11 JA-STYLE ............................................................................................................................................... 14 Cross-cutting Issues ............................................................................................................................... 22 Monitoring and Evaluation ...................................................................................................................... 23 V. LESSONS LEARNED ..................................................................................................... 25 VI. CONCLUSIONS .............................................................................................................. 27 VII. RECOMMENDATIONS ................................................................................................... 29 JA-Style................................................................................................................................................... 29 HIV/AIDS ................................................................................................................................................ 30 APPENDIX A. SCOPE OF WORK .......................................................................................................... 33 APPENDIX B. PERSONS CONTACTED ............................................................................................... 43 APPENDIX C. REFERENCES ................................................................................................................ 51 iv Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 v ACKNOWLEDGEMENTS This evaluation report was prepared for USAID/Jamaica by a team of international and Jamaican consultants brought together by GH Tech. The team is grateful for the preparatory and continuing support provided by that group, especially Prateeksha Alsi and Julie Klement. They also appreciate the support provided by USAID/J, particularly Jennifer Knight-Johnson, project management specialist, who was responsive to the team’s requests and available for consultation throughout the evaluation. We also thank Leigh Shamblin, director of the Office of General Development, who gave the team insight and encouragement. We appreciate the assistance of JA-STYLE, which accommodated our requests for interviews and helped us with information, scheduling, and transportation. Finally, we are most appreciative of all the time given to us by the many public, private, and nongovernmental organization staff and program participants, who gave us the information and insights for our evaluation and whose commitment to adolescent reproductive health and healthy lifestyle activities will continue to make a difference for Jamaica’s young people. vi Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 vii ACRONYMS ARH Adolescent reproductive health BCC Behavior change communications BSN Bachelor of Science in Nursing CBD Community-based distribution CBO Community-based organization CCPA Child Care Protection Act CoP Chief of party CSW Commercial sex worker FBO Faith-based organization HFLE Health & Family Life Education HIV Human immunodeficiency virus HL Healthy lifestyle IAP Impact analysis plan IPR Interpersonal relations IR Intermediate result JA-STYLE Jamaica’s Solution to Youth Lifestyle and Empowerment J-YAN Jamaica Youth Advocacy Network LoPP Life of project plan M&E Monitoring and evaluation MOE Ministry of Education MOH Ministry of Health MSM Men who have sex with men NCDA National Council on Drug Abuse NFPB National Family Planning Board NGO Nongovernmental organization OAT Organizational assessment tool OC Oral contraceptive PEAS Policy environment assessment viii Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 PMIS Project Management Information System PSA Public service announcement PWG Policy working group RH Reproductive health SDC Social Development Commission SOW Scope of work SRH Sexual and reproductive health STI Sexually transmitted infection TA Technical assistance TFGI The Futures Group International TOT Training of trainers URC University Research Co. UWI University of the West Indies USAID United States Agency for International Development VCT Voluntary counseling and testing WRHA Western Regional Health Authority YAB Youth Advisory Board YFS Youth-friendly services YMCA Young Men’s Christian Association Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 1 EXECUTIVE SUMMARY An evaluation team was assembled to review the Jamaica Adolescent Reproductive Health (ARH) Activity supported by USAID/J, covering 2000–07. The focus of the evaluation was an assessment of two major projects: The Adolescent Reproductive Health Project (2000–04), known as Youth.now, and the Jamaica Adolescent Healthy Lifestyle Project (2005–09), JA-STYLE. The team was also asked to prepare recommendations for using HIV/AIDS funds to support youth and ARH. The team looked at the design, strategies, implementation, and results of both projects, giving particular attention to intermediate results (IRs), cross-cutting issues, and thematic areas. The team reviewed project documents and publications and materials from USAID, the Ministries of Health and Education, and other public agencies. They interviewed informants from USAID/J, ministries, donor agencies, project contractors, former and current project staff, national and community partners, grantees, youth-friendly services (YFS) site personnel and other providers, and youth groups. Information was obtained through face-to-face and telephone interviews, focus group sessions, and site visits. Jamaicans under age 20, who constitute about 20 percent of the population, face significant reproductive and other health risks, particularly HIV/AIDS and substance abuse. Many societal realities account for the RH and HIV risks, especially early initiation of sex (too often forced); multiple sexual partners; insufficient awareness and skills to protect their health; and a lack of information and services to address their needs. As substance abuse becomes a more serious problem among Jamaican youth, it factors into other risk-taking behavior. Roughly one in 10 Jamaican youth is subjected to physical and sexual abuse. USAID/J has provided significant support to ARH in Jamaica, primarily by funding two major projects in support of its strategic objectives for health, although smaller grants also contributed to meeting these objectives. Between 2000 and 2004, $12 million was allocated to support Strategic Objective 3: Improving the reproductive health of youth. The Futures Group International (TFGI) received $8.7 million for Youth.now, which was designed to address three IRs related to increased access to RH/HIV services, improved knowledge and skills, and supportive policies and guidelines. For 2005–2009, $14.9 million was authorized, with an original award of $10.5 million to University Research Co. (URC) for JA-STYLE in support of Strategic Objective 11: Improved health status among youth and vulnerable target groups. The IR on increasing healthy behaviors among youth was to be achieved through four sub-IRs: expanding access to YFS; implementing supportive national policies and guidelines; improving knowledge, attitudes, and skills; and increasing community support and involvement. Although the project was to have operated for five years, funding was later cut by $4 million and duration by 1.5 years because Mission funding was cut. USAID/J’s strategy and approach were to improve ARH and foster healthy lifestyles among youth by supporting the Ministry of Health (MOH), especially by identifying gaps and areas of needed support. The organization also emphasized collaboration with other sectors and the sustainability of project initiatives. In the second period, the mandate for JA-STYLE was broadened to cover substance abuse and violence in addition to ARH and HIV/AIDS prevention. However, the public and the institutional environment both presented challenges for implementing these projects. Jamaicans are hesitant to confront sensitive cultural issues, especially those related to sexuality. Institutionally, there is not yet a solid consensus on how services should be delivered to young people, and where ARH would be best situated within the MOH. 2 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Among the findings about Youth.now’s performance the team identified were these: • The Youth.now program was strategically focused, ambitious, and consistent with MOH objectives, and target groups were well identified. • Although many intermediaries (such as service providers and parents) were involved, direct reach to youth was limited. • A curriculum and other ARH materials introduced into nursing and midwifery schools have been sustained. • Mass media campaigns were stratified for different ages and reinforced safer sex behaviors. • Solid partnerships were established with churches through Christian Family Life Education training and related efforts. • The Policy Working Group (PWG) promoted positive policy efforts. Some of the major findings for JA-STYLE were that: • JA-STYLE was designed as an ambitious program with a broad scope and challenging operational plans. • The majority of activities are not reaching vulnerable groups and those in greatest need. • JA-STYLE does create visibility through media and public events that reach a large number of people. • Because coordination between project components (IRs) is not sufficient, activities within the project are not well integrated. • JA-STYLE’s support of improved interpersonal relations (IPRs) in both clinical and nonclinical services responds to an identified need and complements MOH priorities. • The intended focus on ARH, especially making contraception (including condoms) available, has been significantly reduced in JA-STYLE activities. • Media materials, especially the radio show (“Outta Road”) and drama (“Curfew”), are professionally produced and positioned to reach a large audience. • The budget cuts significantly impaired JA-STYLE’s ability to use the small grants program to mobilize communities effectively. • Some partners and grantees not working in ARH before collaborating with JA-STYLE are now engaged in this area in a meaningful and sustainable way. • The Jamaica Youth Advocacy Network (J-YAN) is a vital, active, and potentially sustainable force that can effectively address youth issues in policy and advocacy nationally. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 3 The team’s major recommendations for what JA-STYLE could best focus on before the project ends are these: • The project should focus on interventions that have the widest reach and the highest probability of sustainability. • USAID/J should have more dialogue with the MOH to achieve a consistent vision and feasible approach to delivering ARH services. • JA-STYLE’s approach should be altered to enhance, rather than restrict, adolescent access to condoms. • The capacity and sustainability of organizations still receiving grants should be reinforced. • There should be a plan for organizational continuation only if an effective, cohesive operational entity could advance ARH and youth development. The team was also asked to recommend ways USAID/J could most effectively address the critical issue of preventing HIV in youth. These are the major recommendations, which depend on Jamaican law, donor policy, and available resources: • Ensure that the new Ministry of Education (MOE) curriculum on Health and Family Life Education for grades 7–9 is implemented as envisioned, complete with condom discussions and demonstrations. • Explore promotion of policies that would help make condoms available in schools. • Teach males about condom use at earlier ages. • Scale up HIV testing, including voluntary counseling and testing (VCT), and target the most vulnerable populations, such as men who have sex with men (MSMs), commercial sex workers (CSWs), and disabled youth. • Implement a public education campaign to address social norms related to sexuality and gender, including risk-taking, sexual abuse, acceptability of condoms, and linkages between violence, substance abuse, and HIV. In conclusion, USAID/J has been a solid supporter of ARH and healthy lifestyles since 2000. Its support to ARH and related lifestyle activities was concentrated on two projects, Youth.now and JA-STYLE, and it worked in close collaboration with the MOH. USAID/J funding has enabled models to be tested, brought important issues to the surface, and spotlighted the importance of monitoring and evaluation (M&E). It has also helped to emphasize the importance of policy issues and to identify partners that could help to support healthy adolescent lifestyles. One aspect of Youth.now’s work was to test models for delivering YFS; these proved to be generally unsuccessful because they faced many barriers. Other areas were thought to be more successful, including training of key intermediaries (such as providers and parents) in ARH; institutionalizing pre-service training in nursing schools; a mass media campaign; policy development related to access of minors to contraceptives; and establishment of a national youth policy. While Youth.now’s record may have been mixed, several of its initiatives have been sustained, and its visible outreach to the public and to professionals has deepened understanding of ARH issues. JA-STYLE carried out a great number of activities, covering media exposure; development and dissemination of an IPR program; establishment of an active and effective youth advocacy network; small grants that helped a number of community-based organizations to carry out important work (including 4 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 some that helped others to do so); initiating collaborative activities in violence-prone communities; and initiating capacity-building activities with community groups and NGOs. That many of these activities did not appear to last beyond the funding or were not sufficiently integrated or focused to effect change is due mainly to cuts in program duration and funding, but there was also a lack of strategic planning early on. Even with its brief tenure, however, some activities could be sustainable; among them are the IPR program (already partially adopted and with certification pending) and the inclusion of ARH by some grantees, such as the 4H and possibly the J-YAN should it attract sponsorship. The MOH as a prime collaborator on these projects reported mixed opinions to the team. Staff expressed differences about the desirability of expanding JA-STYLE’s issue areas and about how much collaboration with them actually took place on this project. Several unresolved issues need to be sorted out before action on ARH or healthy lifestyle agendas can proceed effectively: the conflict between the Child Care Protection Act and the policy on making contraception available to youth; how best to approach delivery of ARH services; and where ARH should be situated within the MOH. All of these seem resolvable, especially the ARH services issue, which has become too polarized between separate services for youth and the same services for all, when in fact some feasible approaches lie between the two extremes. Among areas USAID/J would most need to address in the future are working with the MOH to sort out the unresolved issues on ARH and promoting a change in social norms related to sexuality and gender. Failure to address these issues will slow progress in areas like early pregnancies, multiple partners, transmission of HIV and sexually transmitted infections (STIs), substance abuse, and violence. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 5 I. INTRODUCTION SCOPE OF WORK An evaluation team was assembled to review the work of the Jamaica Adolescent Reproductive Health (ARH) Activity supported by USAID/J from 2000 through 2007. The evaluation focused on two major projects funded: the Adolescent Reproductive Health Project (2000–04), known as Youth.now, and the Jamaica Adolescent Healthy Lifestyle Project (2005–09), known as JA-STYLE (Jamaica’s Solution to Youth Lifestyle and Empowerment). The team was also asked to recommend how future HIV/AIDS funds might best be used to support youth and ARH (see Appendix A, Scope of Work [SOW]). However, given the broad scope of the SOW and the time and personnel limitations of the evaluation, USAID/J agreed that the team need not review any other grants and activities. Instead, there would be a general review, to the degree possible, of USAID/J’s support to ARH since 2000. EVALUATION TEAM After some adjustments in personnel, with the concurrence of USAID/J and GH Tech the five members of the evaluation team slightly altered the tasks and responsibilities assigned in the SOW to better suit their skills and expertise. The final members were the team leader/evaluation specialist, Judith Senderowitz; the adolescent services and behavioral change communications (BCC) specialist, Tijuana James-Traore; the community participation specialist, Karla Pearcy-Marston; the policy advisor and local liaison manager, Jennifer Stuart-Dixon; and the youth representative/consultant, Roxanne Johnson. All team members have worked in Jamaica on ARH/HIV activities; two are Jamaicans and another resides in Jamaica. Ms. Senderowitz has served as director of several nongovernmental organizations (NGOs) and as advisor to governments and NGOs; performed evaluations in all developing country regions; created tools for ARH programming; and written widely on ARH projects. Ms. James-Traore has also specialized in ARH; she has significant experience in training, strategic planning, curriculum development, and providing technical assistance (TA) in all geographic regions. Ms. Pearcy-Marston has specialized in ARH, maternal and child health (MCH), and child survival, working with community and refugee groups primarily in Latin America and the Caribbean. Dr. Stuart-Dixon most recently managed the Voluntary Counseling and Testing (VCT) Program for the Jamaican National HIV Program; her areas of specialization are training, counseling, and management in Jamaica and elsewhere in the Caribbean. Ms. Johnson, a recent graduate of the University of the West Indies (UWI) and the team’s youth representative, specializes in media communications and public relations, serving as an intern and youth leader in these areas in Jamaica and St. Lucia. METHODOLOGY The evaluation team looked at the concept, design, strategies, implementation, and results (where possible) of both projects (see summaries in III, below), paying special attention to the IRs, cross-cutting issues, and thematic areas. Because Youth.now had completed its activities nearly three years before, interviewing staff and making visits to active project sites proved difficult. Also, because there had been a mid-term evaluation of Youth.now, the current evaluation put more emphasis on assessing the current USAID/J-supported project, JA-STYLE. For both projects, contracts and life of project plans (LoPP) were used as a basis for project intentions, supplemented by later planning documents. For JA-STYLE the contract modification was also used, given the cutbacks in time and resources the project faced. The team interviewed informants from USAID/J; several central and regional departments of the Ministry of Health (MOH); the Ministry of Education (MOE); the Ministry of National Security; donor agencies; University Research Co. (URC); other JA-STYLE partners (contractors); former Youth.now staff; JA￾STYLE staff; national partners; grantees; representatives of violence-prone communities, YFS sites, and 6 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 other providers; JA-STYLE’s two youth groups; and young people at several sites (see Appendix B, Persons Contacted). Most of the information was obtained through face-to-face interviews, though a few telephone interviews were necessary. Focus group sessions were held with young people. The team also conducted site visits to observe activities and settings for providing services and educational offerings. In addition, the team reviewed a variety of other project documents, including annual reports, research studies, evaluation reports, policy assessments, grant program guidelines, performance monitoring plans, and program materials like job aids, curricula, advocacy tool kits, radio program segments, public service announcements (PSAs), and the JA-STYLE Highlights newsletter. They also studied materials from USAID, the MOH, the MOE, and other public entities, such as demographic studies, policy reports, and strategic planning documents. Where available, the team examined materials from grantees of both projects (see Appendix C, References). Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 7 II. BACKGROUND Jamaicans under 20, who constitute about 20 percent of the population, face significant reproductive and other health risks, including HIV/AIDS and substance abuse. As both perpetrators and victims, they also face the consequences of high levels of violence in their communities. Many societal realities underlie the RH and HIV risks for young people. Early initiation into sex (too often forced), multiple sexual partners, insufficient awareness and skills to protect their health, and a lack of information and services to address their needs all contribute to negative RH outcomes. Sexual initiation occurs on average at 13.5 years for young men and 15.8 years for young women, according to the Jamaica Reproductive Health Survey of 2002–03—replicating results of the 1997 survey. Young women of higher socioeconomic status with more education tend to delay sexual initiation longer than those of less wealth and education, but even among the more educated the age at which sex is initiated is declining.1 Contraception at first intercourse increased for young women under 25 from 42.7 percent in 1993 to 55.7 percent in 1997 and to 67.3 percent in 2002. The older a young woman is when she begins sexual intercourse, the more likely she is to use contraception from the beginning. Contraception at first intercourse for young men has also increased, but the percentages are lower, from 21.6 percent in 1993 to 43 percent in 2002. As with females, the older the male is at first intercourse, the more likely he is to use contraception. Major reasons given for not using contraception from the beginning include not expecting to have sex (52.5 percent for females and 37.1 percent for males); not knowing any methods (9.5 percent for females and 30.5 percent for males); and not being able to get a method (about 12 percent for both females and males).2 According to a 2001 survey, 73.7 percent of those aged 15 to 19 and 9.5 percent of those 10 to 14 report being sexually active, and while 86 percent acknowledged that pregnancy and STIs are risks of unprotected sex, only 12.6 percent considered themselves personally at risk.3 In fact, more than 75 percent of pregnancies among 15–24-year-olds are unplanned, and about 40 percent of Jamaican women have had at least one child before age 20.4 Young people are very much at risk for STIs. The number of HIV cases continues to increase, and certain risky behaviors seem resistant to change. For example, 25 percent of men and 34 percent of women do not use a condom with nonregular partners. In addition to failure to use condoms with multiple partners, the main risk factors for HIV transmission are transactional and commercial sex and unprotected early sexual initiation.5 The Jamaica Reproductive Health Survey found that 20.4 percent of those 15 to 19 had experienced forced sex.6 1 CDC, 2003, “Highlights from the Jamaica Reproductive Health Survey, 2002–2003,” www.cdc.gov/reproductivehealth/Surveys/Jamaica.htm. Accessed November 2007. 2 Ibid. 3 Hope Enterprises Ltd., 2001, “Report of Adolescent Condom Survey, Jamaica 2001,” Commercial Market Strategies Project (USAID). 4 National Centre for Youth Development (NCYD), Ministry of Education, Youth & Culture, 2004, “National Youth Policy: Jamaican Youth Sharing the World.” 5 J. Peter Figueroa, 2007, “Prevention Review 07,” PowerPoint presentation at the Annual HIV/STI Control Programme Retreat, October 17, 2007, Ocho Rios, Jamaica. 6 CDC,. Op.cit. 8 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Substance abuse is a growing problem among Jamaican youth; it has a negative impact on sexual decision-making, personal safety, and overall health. According to a 2006 national survey, the drugs used most frequently by people aged 11–19 are alcohol (70.4%); solvents and inhalants (28.2%): cigarettes (27.4%); marijuana (ganja) (24.1%); and beady (nicotine and tar) (14.0%). Ecstasy appears to be an emerging problem. Although 30 percent of students in this survey reported beginning drugs at age 11 or 12 years, it is of significant concern that 26 percent of children 9 and younger have used illegal drugs. Another emerging trend is that the traditional disparity of substance use by gender has narrowed, a departure from previous studies that showed a higher prevalence of use for males than females. The 2006 study found, for instance, that within the 30 days before being interviewed, the prevalence of illegal drug use was 17.3 percent for females, versus 15.7 percent of males.7 Physical and sexual abuse affects roughly one in 10 Jamaican young people.8 Concerns were expressed about rape, incest, and sexual assault. In a 2001 study of 12–16-year-olds in Clarendon, 20 percent reported carrying at least one weapon, and 9 percent reported being threatened or injured with a weapon at school in the 12 months preceding the survey.9 From the late 1990s to the early 2000s, violence resulted in more than 1,000 deaths annually in Jamaica; the rate for 2002 was 45/100,000. The homicide rate in 1991 was 24/100,000; by 2001, it was 44/100,000, an increase of 83.3 percent. 10 7 National Council on Drug Abuse (NCDA). 2006. “Report, National School Survey 2006 Jamaica”, abridged version. 8 NCDA. Op cit. 9 Hope Enterprises Ltd. 2001. “What Protects Teenagers from Risk Behaviours? Applying a Resiliency Approach to Adolescent Reproductive Health in Jamaica.” Change Project (USAID). 2001. 10 “Violence Prevention Alliance – Jamaica Chapter Launched – November 10, 2004.” http://www.who.int/violence_injury_prevention/violence/activities/vpa_jamaica.pdf. Accessed November 2007. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 9 III. USAID/J ASSISTANCE TO ARH SINCE 2000 USAID/J has provided significant support to ARH in Jamaica, primarily by funding two major projects in support of its strategic objectives for health, although smaller grants that also contributed to these objectives. Between 2000 and 2004, a total of $12 million was authorized. Of this, $8.7 million was awarded to The Futures Group International (TFGI), on behalf of the MOH, and collaborating partners Margaret Sanger Center International and Dunlop Corbin Communication to address Strategic Objective 3: Improving the reproductive health of youth. Additional grants were awarded to groups conducting tasks related to BCC, policy development, YFS standards, and studies to help guide the project in its major areas of work. TFGI’s project, Youth.now, was designed to address three IRs: 1) increased access to RH/HIV services, 2) improved knowledge and skills, and 3) supportive policies and guidelines (see IV below for findings related to each IR). Youth.now describes its emphasis as to “prevent unplanned pregnancy (especially first pregnancy) and STIs and, in the process, to also address the related issues of sexual attitudes and behaviours, gender relations and gender-based violence as these latter two impact on adolescent reproductive health.” Youth.now’s major strategies were • the development of service delivery models for youth that could be considered for replication • piloting community-based distribution (CBD) projects for condoms and low-dose oral contraceptives (OCs) • training of ARH service providers and others (parents, community leaders) who reach youth, with an emphasis on changing attitudes • institutionalizing a teaching package on ARH in nursing and midwifery schools • executing two mass media campaigns targeted at different age groups • mobilizing church leaders through Christian Family Life Education training and other support • training for community advocacy • contributing to the development of the National Youth Policy • conducting policy environment surveys to assess the climate for ARH programming. For 2005–09, $14.9 million was authorized, with an original award of $10.5 million to URC and its three collaborating partners, Advocates for Youth, Health Strategies International, and the Population Media Center. URC’s project to support the Ministry of Health, which became known as JA-STYLE, supported Strategic Objective 11: Improved health status among youth and vulnerable target groups. Increasing healthy behavior by youth was to be achieved through four sub-IRs: 1) expanding access to YFS; 2) implementing supportive national policies and guidelines; 3) improving knowledge, attitudes, and skills; and 4) increasing community support and involvement (see IV below for findings). JA-STYLE’s approach was guided by four thematic areas (RH, HIV, substance abuse, and violence), addressed through adolescent spheres of influence (family, school/church, peer groups/clubs, work), with a youth resiliency model to enhance protective factors as an overarching concept. JA-STYLE was expected to build on the work of Youth.now. 10 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 JA-STYLE’s major strategies were • development and implementation of the Interpersonal Relations Learning Program • establishment of two teen centers • re-establishment of the Policy Working Group (PWG) and continued efforts to establish guidelines for ARH service delivery (both begun by Youth.now) • working with the National Family Planning Board to educate providers on the new policy for minors’ access to contraception • establishment of the J-YAN to train and deploy youth advocates to address a variety of healthy lifestyle issues • development of an Advocacy Toolkit • development of training curricula to address healthy adolescent lifestyles and parenting skills • creation and broadcast of a radio series (“Outta Road”) and a drama/musical (“Curfew”) • creation of job aids for use with adolescents • a grants program to support community groups and build their capacities • support of activities in five violence-prone communities. Although the project was planned for five years, funding was cut by $4 million and duration by 1.5 years because of cuts in Mission funding. This necessarily cut back on many of the activities planned by the project and compromised some strategies that needed time to evolve. For example, the grants program had planned additional rounds of funding for the most successful recipients, but the shorter timeframe did not allow this. Another promising initiative was the collaboration with the National Council on Drug Abuse, which had to be cut short. Cuts in project funding and duration also took a toll on staff morale and continuity. USAID/J’s strategy and approach during this period were to address the improvement of ARH and foster healthy youth lifestyles by supporting the MOH, and especially by identifying gaps and areas where support is needed. It also emphasized collaboration with other sectors and the sustainability of project initiatives. JA-STYLE’s mandate was broadened to cover substance abuse and violence in addition to ARH and HIV/AIDS prevention. Although there has been some public dialogue on sexuality issues, public discomfort with topics such as early sexual debut, multiple partners, and alternative lifestyles, including homosexuality, contributes to continuing taboos on discussion of them and to the persistence of risky behavior—something both projects had to confront. There is little will to allow services for young, sexually active people, despite the risks of keeping them ill-informed and underserved. Public institutions, such as schools and health facilities, have been hesitant to address the realities facing many of Jamaica’s youth. The health sector barrier seems more related to making practical decisions about serving this subpopulation; there is simply no consensus on whether to provide separate services for adolescents or to serve them in the same facilities as everyone else. The MOH is leaning toward the latter position, making services “client-friendly” rather than just “youth-friendly.” A further barrier to advances in ARH programming is the lack of agreement on where an adolescent health focus should “live” within the MOH, or how a multisectoral approach to healthy adolescent lifestyles could be conceptualized and managed. Moreover, recent discussions about the positioning of adolescent health within family health have coincided with institutional changes within the MOH. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 11 IV. FINDINGS YOUTH.NOW Overview Youth.now carried out a strategically focused, ambitious program that was consistent with MOH objectives, and its target groups were well identified Youth.now was focused on improving ARH outcomes through increased the use of quality services and preventive practices. The original plan of testing various models of service delivery was consistent with the MOH strategic objectives, supported by BCC interventions and reinforced by new policies. Intermediate target population groups identified as reaching young people were the five Ps: parents, peers (youth leaders), providers, pastors, and partners (males). Stakeholders understood the ARH/HIV focus of Youth.now: Informants both within government and in NGOs were clear about Youth.now’s purpose and mandate. Also, they identified ways they were able to work with Youth.now and contribute to reaching common goals. Virtually every respondent cited the focus on ARH, and many added that this streamlined focus was instrumental in building partnerships and undertaking activities. Although many intermediaries (such as service providers and parents) were involved, direct reach to youth beneficiaries was limited: Significant numbers (estimated at over 2,000) of intermediaries, such as service providers and peer educators, were trained; however, this achievement did not translate into large numbers of the target audience, youth, being reached. The program design, which concentrated on model service delivery sites, accounted for most of this limitation. Very few sites were initiated, even fewer actually became operational, and fewer still were successful in attracting young people (see below). Peer educators typically gained from their training and involvement, but the reach to their peers was much less effective. Work with the faith-based community, parents, and males was notable but limited to certain churches and community activities. Partners and staff had solid technical expertise: Youth.now staff had significant experience in ARH, program implementation, and research. The Futures Group International, the contractor, was a major NGO involved in RH issues, especially policy concerns. Margaret Sanger International was expert in training and service delivery. Dunlop Corbin Communication, which created the mass media campaigns, is a highly skilled and creative Jamaican firm. Working in collaboration with the MOH and other Jamaican NGOs and community- and faith-based organizations (CBOs and FBOs), the project had considerable experience and expertise available. However, the positioning of Youth.now within the MOH and the lack of consensus on how to deliver YFS compromised implementation of several planned initiatives: Youth.now was placed in the MOH Division of Health Promotion and Protection, which emphasizes health education. However, the project’s main objective was to increase adolescent access to services, a focus that could better have been realized within the Division of Family Health Services. Differences in views within the MOH about how to deliver services, given resource constraints, may have further compromised its work. As models proved less than successful and were not rolled out, concerns developed about how many adolescents were actually being reached. This put pressure on Youth.now to move away from its model site approach in favor of more broad-based training program for providers and other key groups. 12 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 IR 1: Increased access to quality RH and HIV/STI services and preventive practices Youth.now established seven YFS pilot sites, with mixed outcomes, but lessons were learned for future work: Programs have long struggled with identifying the most successful, cost-effective strategies for increasing adolescent use of RH services. The Youth.now pilot aimed to answer questions about what worked through seven pilot sites. The YFS centers were three NGO and four MOH sites that offered a range of services, including pregnancy planning, HIV/AIDS prevention education and counselling, and contraceptive services. Although the results were mixed, the pilot helped advance an understanding of what works and what does not. While the physical facilities for the pilot are still operational, only remnants of Youth.now’s ARH initiatives remain because projects were not able to sustain funding; there was a lack of political will to implement the initiative as envisioned; or activities were not cost-effective given the limited reach. For example, the trailer used as “The HUB” at the YMCA still stands but the Y has been unable to replace the funding provided by Youth.now to support the staff counsellor and social worker. A few sites, including the May Pen Health Centre and the site at Glen Vincent, never quite took off. Even more successful models, such as Balaclava, reached only a small number of youth, leading to questions about whether the impact was worth the investment. Questions must also be raised about whether failures were related to the models themselves or were the result of barriers that might have compromised any possibility of success, such as the lack of sustained funding and political support. In addition, most YFS projects need public support and integration into a web of community referrals and services. A curriculum and other ARH materials were introduced into nursing and midwifery schools and are still being used: Institutionalizing ARH into pre-service training for nurses and midwives is a significant contribution to improving services for youth. The fact that four years later the curriculum and teaching packet are still being used is a testament to the sustainability of this initiative. Although the ARH packet constitutes a separate document, it is fully integrated into the pre-service program. Tutors in six public and private nursing and midwifery schools supported creation of the teaching packet and were trained to teach it. The packet was introduced into the schools in 2003 and has been infused into the curriculum ever since. It is now part of the 12-week course offered to second-year obstetric students and is threaded throughout the midwifery program. Aspects of the ARH curriculum have also been included in the new Bachelor of Science in Nursing (BSN) curriculum. Youth.now was a catalyst, in cooperation with the MOH, in drafting and pilot-testing standards for delivery of YFS: Youth.now pursued the objective of implementing national policies and guidelines that support ARH by collaborating with MOH in drafting “Reproductive Health Policy Guidelines for Health Professionals,” a framework within which providers should offer services to youth. It ensures that young people have both access to information to help them abstain and family planning and RH services should they choose to become sexually active. The guidelines are based on fundamental principles of respect for young people, their right to confidentiality, and the reality that many young people will choose to have sex despite the desire for them to delay initiation. While the guidelines were not formally approved during Youth.now’s tenure, it did build support for them. There were significant efforts to broaden access to condoms and oral contraception through non￾traditional service providers, such as taxi drivers and shopkeepers: Numerous studies and surveys have found that adolescents tend not to visit health facilities unless they are ill and are more likely to get condoms from outside the health sector. Youth.now’s training of nontraditional providers (vendors, taxi drivers, and shopkeepers) offered another important avenue through which young people could obtain accurate information as well as condoms. Capacitating nontraditional providers with information and resources to help adolescent customers was an effort to reach youth where they live, work, and play. However, the participation of these providers diminished over time; the final report of Youth.now states that this approach “requires on-going promotion and marketing, as well as supervision of the agents.” Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 13 Nevertheless, working with nontraditional partners appeared to be a worthy effort, and could help make these approaches more effective in the future. IR 2: Improved knowledge and skills of adolescents and providers Mass media campaigns were stratified for different ages and reinforced safer sex behavior: Recognizing that adolescents are a heterogeneous group, Youth.now created material with messages appropriate to different ages and lifestyles. Messages of abstinence targeted younger adolescents; information about protecting oneself from pregnancy and STIs targeted older, presumably sexually active, youth. This approach was thought to be more effective than a broad-based campaign targeting adolescents as a whole, with little regard for differences within adolescent populations. Sound partnerships were established with churches through the Christian Family Life Education training and related efforts: Faith-based institutions play an essential role in the lives of many Jamaicans. In collaboration with the Margaret Sanger Center, Youth.now designed a Christian Family Life Education curriculum for use in training pastors and religious leaders, modifying Sunday school curricula, and training church youth leaders. The training program reached a large number of pastors, youth leaders, and other church-affiliated people. It is believed that this approach has had a multiplier effect, reaching more church leaders through the curriculum and trained trainers. It has also reached congregants and others in the faith community through integration of its concepts and messages into sermons, Sunday school classes, and church-sponsored events. Significant training efforts were undertaken with identified target audiences: Youth.now training activities reached hundreds of people throughout Jamaica. In fact, some four years after the project ended people still talk about the impact the training had on their understanding of adolescence—a testament to the sustainability of the training activities. The training plan targeted specific audiences and created or adapted materials to address the needs, knowledge, and skills of these audiences. Materials developed by Youth.now were widely disseminated and used: Posters, training manuals, and other materials published by Youth.now were evident in many health facilities and NGOs visited during the evaluation. This demonstrates the continuing relevance of the messages that resonate with youth today. IR.3: National policies and guidelines in support of ARH implemented An instrument for measuring the ARH environment showed positive changes from 2002 to 2004: Youth.now, in collaboration with the Policy Project II, drafted the Policy Environment Assessment (PEAS) tool to measure perceptions of the youth policy environment in Jamaica. Implemented in 2002 and again in 2004 PEAS identified positive changes in the environment in two main areas: political support for ARH, which included support for HIV/AIDS prevention from government and the public, and policy formulation, which spoke to the existence of a youth and an adolescent RH policy. Positive policy efforts were achieved by the Policy Working Group (PWG): The PWG was initiated in collaboration with the MOH and the Planning Institute of Jamaica to address issues that affected provision of services to adolescents. The group drafted “Reproductive Health Policy Guidelines for Health Professionals” that address contraceptive advice, counseling, and treatment for persons under 16 by professionals at health facilities. They were issued in 2004, as were guidelines for the treatment of STIs and HIV/AIDS in adolescents. Youth.now collaborated on establishing the National Youth Policy: Another positive policy effort was with the National Centre for Youth Development, which drafted components for the National Youth Policy, also finalized in 2004. This policy provides a common agenda for effective development of Jamaican youth, building their individual competencies and helping them to grow into successful adults. 14 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 JA-STYLE Overview JA-STYLE was designed as an ambitious program with a broad scope and challenging operational plans: JA-STYLE expanded on the scope and approach of Youth.now, adding a community-based component and two themes, prevention of violence and of substance abuse. It was also intended to expand geographically from Youth.now’s actual scope to cover all the parishes in Jamaica. Violence and substance abuse are not topics traditionally handled by USAID RH-supported projects except for sexual and gender violence. The LoPP calls for an extensive array of activities at the national, parish, and community levels dealing with four process components and four themes. It is not surprising that the plan proved too ambitious for realistic implementation, given that both duration and resources were reduced. While some activities, such as the Healthy Lifestyles curriculum, allowed full coverage and integration of the four themes, others had difficulty in doing so or elected not to. This was especially true for community grantees, because not all the thematic areas were equally central to their work and it was often difficult to integrate new content into projects already underway. As a result, for example, substance abuse seemed to stand alone or was absent from many activities. The conceptual framework and strategy guiding project activities were insufficient: Because the conceptual framework presented in the LoPP fails to deal with interactions between components, it lacks the underpinnings for integration. This resulted in poor coordination and inconsistent linkage of activities to project objectives. Given the project’s complexity, to achieve prioritized outcomes a strategy was needed to tie together the disparate parts in ways that are mutually supportive. The project lacks a unified, cohesive plan for training: Training has been a central part of work across the IRs and has built the capacity of a broad range of youth-serving individuals and organizations. However, the project would have benefited from planning for training across the IRs, rather than developing individualized training plans. Although the modified contract called for elimination of the training plan, the modification came after the project was initiated. A unified plan from the beginning of the project would have allowed for better planning later and for the components to pool resources in order to maximize the use of funds. Staff are energetic, committed, and well connected, and carry out a considerable number of activities: The JA-STYLE staff consists of dedicated and enthusiastic individuals who are committed to their assignments and come to the project with diverse skill sets and positive connections to organizations throughout Jamaica. Each member of the staff seems to be responsible for a large number of activities. Their energy and willingness to work hard are essential assets of the program. The majority of activities are not reaching vulnerable groups and those in greatest need: Most of JA-STYLE’s activities require individual young people to attend an event, go to a center, or join a group that carries out activities supported by JA-STYLE, but most marginalized and vulnerable youth do not join clubs or attend public events. Many of the community activities conducted by JA-STYLE grantees are mainstream activities, such as homework assistance, marching bands, parenting classes, and earning badges. Special efforts are necessary to gain the trust of vulnerable youth and allay their fears. However, J-YAN does stand out as successful in reaching vulnerable youth, primarily though the efforts of some individuals whose connections are used to reach subpopulations involved in violent activities. Also effective is JA-STYLE’s hiring of a disabled young person, a survivor of a violent crime, who reaches out to youth at risk through training in school settings. JA-STYLE creates visibility through media and public events that reach a large number of people: in addition to its mass media broadcasts, JA-STYLE has been successful in arranging TV and radio interviews, print and electronic coverage of events and developments, and significant attendance at extravaganzas and other public events. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 15 There is some ambiguity among professionals and stakeholders about JA-STYLE’s mandate: Several informants, including senior government professionals, were not clear about JA-STYLE’s mandate. Although grantees were typically clear about their own areas of focus, they were not always aware of the four themes or how they worked together. Moreover, some potential grantees understood that they were to receive grants or felt qualified to do so but were not able to reach agreement with JA￾STYLE. Others wanted to work with JA-STYLE in some way but found it difficult to identify areas of mutual concern. The operational philosophy of JA-STYLE appears to impede rather than enhance young people’s access to contraception and methods of RH protection: Some JA-STYLE and grantee staff informed the evaluation team that they are reluctant to support making condoms available for sexually active youth or did not view JA-STYLE’s mandate as including increasing access to contraception. Although JA￾STYLE worked with the National Family Planning Board (NFPB) to help establish minors’ access to contraception, an effort to clarify and enforce sections of the Child Care Protection Act (CCPA) would have the likely effect of reducing access. Management Turnover in JA-STYLE leadership in the early stages contributed to a slow start and decreased opportunities to plan adequately: Although when project operations began in February 2005 the two top positions were filled as identified in the award contract, these individuals did not work out. There were interim replacements for both Program Manager, not permanently appointed until November 2005, and Chief of Party (CoP), not permanently appointed until March 2006. Some activities did begin during this period, but the project lacked leaders to guide major planning. Spending on activities began slowly, but the project then felt pressure to move forward quickly, perhaps sacrificing more thorough planning. Insufficient coordination between IR components meant that project activities were not integrated: Work plans strove to integrate activities, but trying to roll out a complex portfolio across multiple thematic areas and sectors that operate unilaterally was a major contributor to the inconsistent relationships between the IRs. Though JA-STYLE staff outlined activities in the annual work plan for each IR, individual work plans did not coordinate activities across the IRs, and many opportunities for synergies between project components, such as BCC-created linkages to services, were not exploited. Nevertheless, several activities do reveal good integration. For example, the J-YAN, while housed in Policy and Advocacy, organized training activities that included advocacy, BCC, and services. Lack of clear communication and direction and of a shared vision of priorities impedes effective implementation: JA-STYLE staff were not always fully informed about the plans and activities of other IR components or were reluctant to provide information. For example, a number of times during the evaluation process JA-STYLE staff advised the team to talk to the CoP about decisions or program approaches, even when they related directly to their own work. Field staff (among them a regional coordinator) did not know why issues or activities had been featured. A prominent example of a vision that was not shared related to the objective of making contraception more readily available; although the object was clearly stated in the project award and in the LoPP, some staff did not believe the organization should pursue this objective, even with sexually experienced youth. The URC partnership maintains a collaborative process in support of the project but seems to overlook major operating weaknesses: URC and its partners—Advocates for Youth, Health Strategies International, and the Population Media Center—collaborate on project oversight, conduct monthly conference calls to address the project as a whole and emerging concerns, and review all deliverables. In addition to URC, Advocates for Youth and Health Strategies International have staff at JA-STYLE, who report both to their employers and to the JA-STYLE CoP. While this seems to work for most purposes, it may have resulted in reduced strategic and priority-setting functions and left no practical means to 16 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 address weaknesses in project implementation. For example, URC seemed not fully aware of the decrease in ARH emphasis or the practical implications of JA-STYLE’s informal position on condom availability. IR1.1: Expand access to youth-friendly clinical and nonclinical services Training programs have reached a significant number of service providers in the health and other sectors: JA-STYLE has implemented an impressive number of training programs reaching service providers, community leaders, parents, and youth. Training courses have focused on healthy adolescent lifestyles (covering ARH, violence prevention, substance use and abuse, and HIV/AIDS), parenting skills, and IPR. The training has increased participant knowledge and skills and provided a better understanding of adolescents and the issues that affect their lives. Training of trainers (TOT) courses have helped sustain project initiatives. JA-STYLE’s support of improved IPR in both clinical and nonclinical services responds to an identified need and complements MOH priorities: Because clients across the age span have identified the attitudes and behaviors of service providers as a barrier to greater use of services, the need for support in IPR is almost universal. In response, the MOH has chosen to focus on IPR as a way of providing friendly services to all, not just youth. The MOH has embraced the IPR approach and the Western Regional Health Authority (WRHA) has incorporated IPR into supervision of current staff and orientation for all new staff. The intended focus on ARH, especially the availability of contraception (including condoms), has been significantly reduced in JA-STYLE-initiated activities: Although JA-STYLE was designed to support an integrated approach to health adolescent lifestyles, RH was always intended to be an integral part of it. However, there is evidence that the RH component has gradually diminished over time. YFS moved away from ensuring access to clinical services, including family planning and preventive methods, to focus more on counseling. Few program sites make condoms easily available to youth, and in fact, some restrict access by keeping condoms locked up or requiring that youth be counseled before receiving them. Staff ambivalence about condom access seems to be complicating this problem. The project’s focus on abstinence supports the objective of increasing the age of sexual debut: Consistent with the USAID strategic objective of increasing the age of sexual debut, many program activities encourage abstinence among youth who are not yet sexually active. This is also evident in the work of grantees and partners. The design and implementation of activities, especially within the YFS component, do not link young people to services: Where grantees facilitate the use of services by sexually active youth, it is often due to their own initiative rather than the result of deliberate effort by JA-STYLE. For example, the Adolescent Centre in Mandeville offers computers, rap sessions, and counselling, but although it is located on the grounds of the Manchester Health Centre, there is no real mechanism for linking youth who visit the YFS site to the health facility—no signs, materials, or publications direct youth from the site to the health facility, and vice versa. There is also no evidence of the link documented in any of the data collected, although that was a recommendation of the March 2007 assessment. It is in fact very likely that youth attend the center without ever visiting the health facility. The assessment of the center further supports this finding by noting, “There was no evidence that HIV testing was encouraged for sexually active adolescents, unless there was a specific reason to be concerned about HIV, such as repeated STIs. There was also no evident referral to the family planning section of the clinic, except when a young woman needed to determine if she were pregnant.” The report goes on to say, “The clinic and the Youth Centre appear to be quite separate from one another. While it may provide some added adolescent-specific services and make young people more inclined to seek health care assistance, it may also pull adolescents further from the standard health care services and widen the gap between standard services and youth services.” Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 17 The situation of the Youth Centre in Port Antonio, in the parish of Portland, is similar. JA-STYLE supported enhancement of the center by providing furniture and a sound system for the “chill” area, and counselling services will be offered on site by a volunteer. However, aside from an informal relationship between the JA-STYLE short-term intern, the center, and the community health facility, there appears to be no effort to ensure that adolescents are linked to needed clinical services. Other component areas (advocacy, community mobilization, and BCC) do not work synergistically with YFS to link young people to services: Just as more can be done to link young people to services within YFS, more could also be done to ensure that youth know what other services are available and where. There have been several missed opportunities to provide youth with this information. For example, a tagline could be added to the end of each “Outta Road” radio episode to direct youth to information sources and services related to the topic presented. Also, all materials should follow the model of the job aids using BCC that list phone numbers where youth can call for additional information or assistance. Few services or activities, except for those targeting the disabled, are specifically directed to reaching more vulnerable populations: JA-STYLE reports that since many of its grantees are working with hard-to-reach, vulnerable populations, by extension it too is reaching these populations. However, this does not demonstrate that JA-STYLE is making intentional efforts to reach these groups. Moreover, when what JA-STYLE has funded through grants is examined, resources have not been directed to reaching more vulnerable youth. In fact, the activities that were funded seemed instead to reach a more self-selected group of adolescents. The work JA-STYLE has done with pharmacists has potential to increase condom access for sexually active youth and to be scaled up: Local pharmacists have been trained in adolescent-friendly health services. They have also been provided with materials for display and distribution to young clients. Pharmacists contacted by the evaluation team reported that they are using the skills and information they acquired and that it is having a positive effect on their adolescent clients. The IPR program materials and method of delivery lack clarity: The IPR materials clearly meet a need identified in the field. They serve as a useful tool in improving services for all and in advancing services for adolescents. It is therefore important to ensure that the IPR approach is sustained beyond the life of the project. That will require a clear methodology for transferring the IPR approach, and more clarity about how target audiences are expected to use the IPR training materials and what their role would be when they return to their own organizations. IR1.2: Support implementation of improved national policies and guidelines JA-STYLE has successfully institutionalized the Policy Working Group in the MOH: The PWG did not continue after Youth.now ended, but JA-STYLE revived it and institutionalized it in the MOH. To ensure PWG’s continuity, JA-STYLE drew up terms of reference, which are now being approved by the Permanent Secretary. Collaboration with stakeholders has brought their buy-in on policy, advocacy, and youth development issues: The policy work undertaken by JA-STYLE has been a collaborative process. The efforts to clarify aspects of the CCPA have involved dialogue with representatives from the Child Development Agency, the National AIDS Committee, and the NFPB, among others. The NFPB further collaborated with JA-STYLE and received funding to train providers on the access to contraceptives for minors policy as well as to how to disseminate the policy. Another collaborative effort is JA-STYLE’s work in developing the multifunctional team in the MOH. The team reports to the interministerial committee within the MOE and occupies two seats on that committee. This work helps to institutionalize health into the MOE’s national youth response. The 18 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 national youth response addresses adolescent and youth policy issues across all ministry and government agencies JA-STYLE is working to institutionalize standards and guidelines within the MOH for serving adolescents: Notwithstanding the lack of consensus within the MOH on a service delivery approach for adolescents, JA-STYLE continues to work to institutionalize standards and guidelines for provision of services to adolescents. The recently appointment of a chief medical officer with an interest in adolescent health may bring about some positive steps in this area. This is an area where JA-STYLE recognized the need to institutionalize work begun by Youth.now for sustainability. J-YAN is undertaking positive policy and advocacy efforts: J-YAN is a vibrant youth-led voluntary group that since it began its work in August 2006 has with JA-STYLE support made important contributions to policy, advocacy, and public education. J-YAN has engaged many youth through two major advocacy-training efforts, has formed a steering committee, and now involves approximately 140 young people across the island. Its continuing contribution to policy and advocacy is evident in the revision of the Advocacy Toolkit and the training of J-YAN members as trainers in advocacy. The toolkit is an excellent document for building youth advocacy skills and the capacity of youth leaders. J-YAN has secured two seats on the International Youth Leadership Council, an Advocates for Youth project that empowers youth leaders to advance the RH and other interests of their peers. This is a momentous contribution to advocacy because the issues of Jamaican youth are now being brought to an international forum. Currently, J-YAN targets four areas for advocacy: disability, violence, SRH, and education (with recent additions of care and protection and vocational training). Current efforts to clarify aspects of the CCPA may compromise adolescent access to contraceptives: The CCPA presents contradictions in light of the policy guidelines for the provision of contraceptives to minors. JA-STYLE’s work is geared toward reaching a clear consensus on how to treat sexually active minors who seek contraceptive services. However, with a mandate that seeks to expand access of contraception to minors, it must be careful to ensure that the policy protects the adolescent at risk of coerced sexual activity yet does not act as a deterrent to adolescents who choose to be sexually active and responsibly seek services. JA-STYLE’s efforts in this area should at the very least be trying to push exemptions to guarantee access for adolescents 16 years old and older. Otherwise, the fallout from clarifying the CCPA could deter adolescents in general from seeking services and further limit their access. IR 1.3: Improved knowledge, attitudes, and skills Materials developed by JA-STYLE seem to be widely used: JA-STYLE has made tremendous efforts to distribute its materials. They were visible in several locations visited as part of the evaluation process. Many informants reported using curricula, brochures, posters, and other materials published by JA￾STYLE in their daily work. Furthermore, the dissemination plan ensures wide distribution of materials at the end of the project. Media materials, especially the radio show (“Outta Road”) and drama (“Curfew”), are professionally produced and poised to reach a large audience: All too often materials developed for adolescent programs lack the quality and appearance that would appeal to young people and that can compete with the many other messages they receive from the print and electronic media. However, that cannot be said for much of what JA-STYLE has produced. Because the professionally produced radio spots and drama have wide appeal, they have the potential to live well beyond the life of JA-STYLE. In addition to disseminating its own materials, JA-STYLE has also facilitated use and distribution of materials developed by other organizations: JA-STYLE has done much to disseminate information from other programs and services, such as Youth.now posters, HIV/AIDS posters from the MOH, and materials from the NFPB, at major events like youth extravaganzas that reach thousands of young people. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 19 There appears to have been some duplication of materials, though a broader scope and possibly differing audiences might be a justification: Several informants stated that some materials published by JA-STYLE duplicated existing materials. Curricula that were mentioned included a HOPE Worldwide Parenting Manual and the MOH Life Skills Resource Manual: towards a Healthy Lifestyle manual. It was agreed that JA-STYLE should have reviewed these documents before putting out its own. Although JA￾STYLE curricula were often either directed to a different target audience or integrated the four thematic areas, other curricula could have been adapted to reduce development costs. Young people have been influenced media activities and materials: JA-STYLE has made significant efforts to include young people in all aspects of the development and production of BCC materials. The BCC component has used Youth Advisory Board (YAB) input for its products and activities and has established a listening group for the “Outta Road” series. Youth in these groups were able to identify cases where their own inputs influenced the final product, confirming that they were more than tokenism. The involvement of youth in JA-STYLE is an important model for other youth programs, though enhancements are needed to improve upon current efforts. For example, the plan for establishing additional listening groups should be pursued to ensure greater representation of youth from different backgrounds and settings. In addition to recalling what they have heard earlier, groups should listen to the episodes during the sessions to give direct feedback on content and messages. There are good examples of youth from high-risk situations being involved in innovative ways: JA￾STYLE has demonstrated a commitment to involving youth from high-risk situations in several ways. Its staff include young people who represent vulnerable groups, such as the disabled and those who have been involved in violent activity; several grantees, such as Children First and Kids Camp, now employ young people who were once beneficiaries of their programs. IR 1.4: Increase community support and involvement Budget cuts significantly impaired JA-STYLE’s ability to mobilize communities effectively through the small grants program: The reduction in JA-STYLE’s budget resulted in major disappointments for grantees, who were expecting support to continue over several years. It also hampered the project’s plan to assess grantee performance over time and provide support to the most promising groups. More important, it impaired JA-STYLE’s ability to work closely with each grantee organization on strategies to mobilize communities in support of healthy lifestyles and ARH. The budget cuts limited capacity￾building efforts with grantees, most of which lack basic skills in areas like accounting and financial reporting, training, fundraising, and youth involvement. JA-STYLE is now working with the few remaining grantee organizations still completing their end-of-project reporting. The community engagement process within the small grants program did not adequately identify some critical issues: As part of JA-STYLE’s efforts to initiate the small grants program, consultations were held in eight parishes with various stakeholders to “identify priority issues specific to each parish; and consultatively select issues to be addressed through the grants mechanism to increase healthy lifestyle behaviors of adolescents.” JA-STYLE provided basic criteria, such as organizational viability and demonstrated interest in working with youth. It then drafted requests for proposals for each parish based on the priorities laid out by the CBOs, FBOs, and NGOs involved in the consultations. Although the process was positive in that it allowed communities to identify their own needs, it appeared to lack strategic guidance from JA-STYLE to ensure that some key issues would emerge. For example, critical issues affecting youth RH, such as sexual abuse and domestic violence, did not get specific attention during the consultations and thus were not included in program activities, except as part of parenting programs, on the assumption that these topics related only to poor parenting skills. It would have been beneficial for JA-STYLE to facilitate parish-level dialogues on sexual and gender-based 20 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 violence, which is inextricably linked to the complex RH realities for youth and relates directly to the project’s concern about the crosscutting issue of violence. Some partners or grantees that had not worked in ARH before collaborating with JA-STYLE are now doing so in a meaningful and sustainable way: One noted success of JA-STYLE is that it mobilized organizations not previously working in ARH issues to do so. 4H, Westmoreland Association of Clubs, Kids Camp, Girls’ Brigade, and the Social Development Commission (SDC) are among grantees and partners that now appear to have a long-term commitment to engaging with youth on ARH issues, even after their grants or funding support are closed out. The following examples illustrate this positive engagement in ARH: • 4H conducted training sessions with youth and parents covering healthy lifestyles and sexual decision-making. Now it appears to have systematically integrated ARH training and activities both within local chapters and at the national level. • The Westmoreland Association of Clubs in Petersfield, which had previously not worked with youth, started a Football for Education program for unattached boys. Its healthy lifestyle sessions on topics such as prevention of STIs have integrated both youth and their parents and it continues to focus on ARH issues even since completion of its JA-STYLE grant. • Although ARH was not totally new to Kids Camp, it became more committed to RH, youth-adult partnerships, and participatory methods. • A significant success occurred with the Girls’ Brigade, which describes itself as “Helping Girls and Young Ladies to Achieve Their Full Potential.” This group moved from being hesitant to deal with condoms to obtaining a grant for increasing access to condoms through social marketing. • SDC trained Community Development Officers in several parishes on ARH using the Adolescent Healthy Lifestyles curriculum. Now SDC includes adolescent healthy lifestyles topics in its general community development activities. Capacity-building efforts with grantees were mixed, but for the most part, they were limited, mistimed, and not necessarily responsive to each organization’s unique needs: A critical component of the small grants program is to strengthen the capacity of participating organizations so that they can sustain ARH and healthy lifestyle activities beyond JA-STYLE funding. There are some positive examples of JA-STYLE supporting organizational capacity-building efforts among grantees, but efforts in this area have been limited, apparently due in part to USAID budget cuts, which significantly curtailed the grants program but also in part to JA-STYLE’s mixed response to the unique issues and needs of specific organizations. JA-STYLE used its Organizational Assessment Tool (OAT) when a grant was awarded to assess the technical strengths and weaknesses of grantees and establish a baseline for measuring capacity. The OAT was repeated at the end of the grant to ascertain whether any changes resulted from project interventions. Of the 19 grantees assessed, 63 percent (12 of 19 organizations) showed some improvement in one or more capacity areas measured. Many JA-STYLE efforts in capacity-building involved large group trainings conducted for a number of organizations (e.g., nine or more grantees) on technical topics such as BCC, community mobilization for HIV/AIDS, and ARH. Although these trainings were no doubt helpful to many grantees, JA-STYLE appeared to be less focused on identifying and strengthening each organization by addressing areas that are potentially more vital to sustainability, such as board development, financial management, or fundraising. One training on financial management was offered to a large number of grantees, but the results were mixed because many of the participants had already closed out their grants. During evaluation interviews Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 21 with eight grantees, several organizations expressed significant disappointment about the lack of TA from JA-STYLE to build skills in areas like report writing, youth leadership development, fundraising, and general management. Some grantees provided mentoring and cross-fertilization of technical expertise to other organizations: One positive result of the small grants program is that it brought together groups that had not previously coordinated or shared resources. Examples are five grantees in Clarendon and Manchester who still meet periodically to share technical expertise. Children First and Women’s Centre, which are more established organizations, have also provided TA to other groups. The involvement of the SDC, a partner with national reach, was strategic to facilitating community involvement in ARH: Engaging SDC, a government agency in charge of community development, was thoughtful and effective; it facilitated involvement in youth and healthy lifestyles issues on a national level. Before receiving financial support from JA-STYLE, SDC’s focus was on long-term community development. Since becoming a partner, it has expanded its work to involve youth on a national level, integrating ARH issues through healthy lifestyles training as part of community development. These new efforts have enabled SDC, which is already well positioned to work with community gatekeepers, to engage communities on the grassroots level quickly and effectively on behalf of youth. The process of designating the five violence-prone communities seems to have been collaborative and strategic: To initiate its work on violence prevention and to select the first four communities for funding, JA-STYLE hosted a meeting with representatives from the MOH, Peace Management Initiative, the Ministry of National Security, the Violence Prevention Alliance (associated with the UWI), USAID, and the Prime Minister’s Office. Using multiple criteria, such as demonstrated leadership ability within the community, government designation as a “violence-free zone,” and the existence of potentially supportive agencies in the area, the stakeholders attending this meeting designated four target communities. This process involved key collaborators and instilled a sense of ownership beyond JA￾STYLE’s planned funding to the communities. Later a fifth community was added at the request of USAID. Yet support to the five violence-prone communities was not implemented in a way that advanced project objectives or sustainability: One shortfall of JA-STYLE’s support to the designated communities is that the grant-funded activities were not well linked to improving the use of quality services within the four thematic areas (RH, HIV/AIDS and STIs, substance abuse and prevention, and violence prevention). For example, Flanker was funded for marching band uniforms and sporting gear, and Browns Town for drum instruments and netball uniforms associated with after-school programs. The weak connection between ARH, violence prevention for the most vulnerable youth, and what JA-STYLE funded in these communities made it difficult to advance project objectives in a meaningful or sustainable way. An additional problem was that violence did not appear to be clearly defined by JA-STYLE and the grantees, and omitted such important areas as sexual and gender-based violence. Overall, substance abuse mitigation efforts were minimal and not well integrated into project activities: JA-STYLE had designated substance abuse, along with violence, as a thematic area to be integrated into general project efforts. It is commendable that JA-STYLE identified it as a critical issue negatively affecting youth sexual decision-making, HIV/STI, and violence risks. One noted success in this area was the project’s positive collaboration with the National Council on Drug Abuse (NCDA) on TV and radio PSAs addressing substance abuse. Unfortunately, due to the budget reduction, other activities planned with this partner, such as summer workshops, were cut. Within JA-STYLE’s ambitious operational plans, however, it appears that substance abuse efforts were largely lost, and integration of the theme with other project components was limited. Although substance abuse topics were included in such project materials as the Adolescent Healthy Lifestyles curriculum, these efforts were not generally well developed. In interviews with JA-STYLE staff the evaluation team 22 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 noted confusion and lack of clarity about where substance abuse efforts “live” within the larger project. Moreover, substance abuse did not emerge as an area well integrated within the activities carried out in the violence-prone communities. Sexual and gender-based violence did not appear to be a focus of violence prevention efforts: It is well known that sexual abuse (including incest), sexual coercion, rape, and domestic violence are increasingly troubling issues for youth and directly relate to negative RH outcomes, such as incidence of HIV/STIs and pregnancy. Interviews with informants associated with JA-STYLE’s violence prevention and grantee organizations did not identify sexual and gender-based violence as an issue included in their definition of violence to be addressed, except perhaps as a topic touched on in parenting training. This was a lost opportunity. The project could have usefully opened up dialogue on these challenging and traditionally taboo topics. CROSS-CUTTING ISSUES Involvement of Youth J-YAN is a vital, active, and potentially sustainable force that can effectively address youth issues in policy and advocacy on a national level: The J-YAN is a youth-led network that has made impressive progress since its inception in 2006. Its members are from youth organizations across the island, including grantee associations. Training provided by JA-STYLE and other partners has equipped them to advocate effectively for issues important to youth. The network has already undertaken campaigns related to ARH, violence prevention, and care and protection of the disabled, and is involved in efforts to advocate for incarcerated youth through the corrections system. J-YAN was instrumental in revising the Advocacy Toolkit created by Youth.now to include the broadened mandate of JA-STYLE and to update policy developments in Jamaica. An advocacy training manual is also in the making. Currently, the network is working to identify another parent organization or establish an independent NGO so that at the conclusion of JA- STYLE it can sustain its activities. Some activities have strong youth involvement or are nurturing youth leadership: Some JA-STYLE efforts and activities, such as J-YAN and Children First, foster youth participation and the development of youth leadership. Young people were involved in various critical aspects of JA-STYLE and had significant decision-making and implementation responsibilities, even though some of the involvement occurred late in the project. For example, in October and November 2007 JA-STYLE hired two talented young people as program staff. Though recent this action helps nurture youth leadership potential. Young persons were engaged in establishing the Port Antonio Youth Centre and served as writers for “Outta Road”. However, there were other areas where youth leadership seemed weak. For example, the Adolescent Centre in Mandeville and Westmoreland Association of Clubs appear to be led and coordinated predominantly by older people. Young people were involved in the early stages of the project and made some contributions to its development: There was noticeable involvement of young people as participants in consultations at the community, parish, and regional levels; as researchers and script writers for “Outta Road”; as participants in the nationwide project naming competition; and as participants in the study tour to South Africa. However, listening groups for the “Outta Road” radio series were established only after some of the episodes were produced, which may have limited their value in the early stages of production. Also, J￾YAN and the YAB were established after the formative stage of the project. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 23 Community and Parish-Driven Implementation Model The grants program was heavily influenced by the needs of communities and parishes: The approach JA-STYLE took to identify needs in the parishes, communities, and organizations selected helped ensure that the activities funded were consistent with their objectives. As already mentioned, however, the downside was a lack of strategy to ensure a collective result within parishes rather than just in organizations and a missed opportunity for JA-STYLE to offer directive leadership in pushing the envelope to deal with challenging issues like sexual coercion. Intersectoral Collaboration Aside from the MOH host unit, Health Promotion and Protection, and the SDC, JA-STYLE had a mixed record of collaboration with government agencies: There was significant collaboration with the MOH host unit and SDC, but although JA-STYLE reported considerable interaction with various sectors, including democracy and governance, these activities were not identified or brought to the team’s attention during the evaluation. Sectoral staff interviewed by the team reported little productive collaboration. Coordination with Other Donors Collaboration with donors was uneven: Several donors reported little collaboration with the project, although JA-STYLE identifies collaboration with the major donors. Apparently, collaboration with donors was curtailed because the Mission wanted project activities to start up faster. Thus, the collaboration reported appears to have been informal or to have occurred only recently rather than earlier in the project. Involvement of the Private Sector There was some success with tapping the private sector on access to air time, in-kind contributions, and financial support: Among the organizations that supported JE-STYLE activities were Digicel, which gave almost J$1.3 million, and Cable & Wireless, which provided for the sending of text messages for Peace Month. Although these were positive private sector collaborations, they were limited; others could have been pursued to strengthen project activities. Scaling-up Best Practices Mass media activities were implemented but there was no scaling-up of project models: Significant private sector support enabled the radio program to be implemented as planned. IPR orientations (though not models) were scaled up after initiation. The reduced period of operation and lack of a plan to evaluate project models did not allow for further scaling-up. Projects were also challenged by the realities of trying to sustain themselves at the current level of activity. MONITORING AND EVALUATION The JA-STYLE Impact Analysis Plan (IAP) will be unable to attribute results to its efforts because it lacks a baseline measure and will find it difficult to identify uncontaminated control groups: When implemented the IAP will look at impact rather than whether activities were carried out according to plan. The plan itself says “the lack of pre-intervention data [is] perhaps the greatest limitation to JA￾STYLE project evaluation research.” The team was informed that URC was asked not to carry out baseline assessments because these were to be done by MEASURE, but the results were available too late for use as a baseline. The plan to compensate for this gap in data by comparing project participants with control groups will be challenging in Jamaica, which hosts a considerable number of activities in areas covered by JA-STYLE. It will be difficult to identify groups that are comparable enough to qualify as control groups but have not been exposed to the activities of JA-STYLE or another group providing related information, messaging, and BCC opportunities. 24 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Significant data collected by the Project Management Information System (PMIS) from project activities are used selectively for decision-making on project sites and strategies: According to staff reports, data from the PMIS have been used to help communities implement healthy lifestyles, for establishing new advocacy networks, and for targeting the radio serial. It is not clear how data on utilization of health services informed an approach to YFS because little is currently being done in this area beyond counseling. There is no plan to measure the pathway from YFS center attendance to use of clinical services: at the Adolescent Centre at Manchester Health Centre, the team found an array of activities designed to attract adolescents, such as homework study assistance, computer use, games, and physical fitness options. Clinical services that were once provided are no longer offered because of staffing and financial constraints. While the YFS center is on the grounds of the Health Centre, presumably for easy referral, it does not document the number of young people coming to it who need or are referred to clinical services. The monitoring data overseen by JA-STYLE counted only the number of young people who used the Health Centre; it did not track which of these originated in, or were referred by, the Adolescent Centre. Support from JA-STYLE provided furniture and painting to enhance the center’s appeal but did little to increase access to clinical services. Youth had limited involvement in monitoring and evaluation: Youth involvement was not implemented as the LoPP had proposed and there was no systematic training of young people in M&E. However, JA-STYLE reports that youth have been engaged as data collectors for the formative research for “Outta Road” and for conducting a separate, youth-focused PEAS, complementing the more technical PEAS conducted with adult partners. (Naturally, young people were subjects or respondents in M&E-related surveys and assessments.) Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 25 V. LESSONS LEARNED The following lessons learned, both positive and negative, were derived from the team’s assessment of both projects and the transition between the two. They can provide guidance for future programming. • Follow-on programs should build upon predecessor activities as much as possible through systematic analysis of past efforts in order to continue worthy initiatives and avoid duplication of materials or unsuccessful strategies. • A clear transition plan, articulating continuity between the completed and the follow-on project, should be drafted to ensure a smooth and efficient process. • Creating favorable conditions for project sustainability requires meaningful engagement with potential donors and partner institutions starting from the very first stages of the project. • In an ARH project, youth involvement should be fostered and sustained from the beginning to inform project design and implementation and promote youth leadership development. • Staff should be able to articulate clearly the fundamental principles and strategies that guide their work. • An effective approach to implementing a complex, integrated ARH and healthy lifestyles project requires that staff at all levels of the implementing organization collaborate, share resources, and coordinate their efforts. • To address effectively the many social issues that affect adolescent health and wellbeing but are considered taboo—such as gender violence, homophobia, and sexual abuse—advocating for these issues must become part of the national debate. • As an important component of healthy lifestyles, delaying too-early pregnancy should get special attention. • Support from key units in the MOH, such as the Family Health Division, is imperative in implementing activities within its purview. • Resolution of the differing views within the MOH on serving adolescents would help to foster needed service delivery to this key subpopulation. • There is a wide gap between ARH information and service needs and public and professional willingness to meet those needs. • Extant curricula should be reviewed before drafting curricula that might duplicate earlier efforts, as the parenting curriculum does. • Partnerships and collaborations with FBOs and others can be successful in gaining support for policy, advocacy, and youth development. • It is imperative that community efforts with youth are linked to needed services, clinical as well as nonclinical. • While community- and parish-driven implementation models offer opportunities for setting priorities, it is also important that grantees be given strategic direction and guidance to ensure that critical issues are targeted for funding. 26 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 • Meaningful, customized capacity-building efforts are necessary within a community-based small grants program to strengthen grantee organizational and technical skills and facilitate their sustainability beyond the life of the project. • Institutionalization of project initiatives or project-supported components within government structures, such as the PWG and the ARH service delivery standards and guidelines, is necessary for their sustainability. • Funding cuts affect not only project implementation and achievements but also staff morale and motivation. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 27 VI. CONCLUSIONS From 2000 through 2007, USAID/J support to ARH and related healthy lifestyle activities was concentrated on two projects, Youth.now and JA-STYLE, which worked in close collaboration with the MOH. Both projects had ambitious mandates and carried out significant programming, both with public entities and in partnership with NGOs, CBOs, FBOs, and to a lesser extent the private sector. JA-STYLE added substance abuse and violence prevention as themes and community involvement as an additional program component. Youth.now placed considerable emphasis on testing models for delivering youth-friendly services. These proved generally unsuccessful, but there were many barriers to be surmounted: the MOH was not in full agreement about supporting services for adolescents as a separate group; too little was done to mobilize the community in support of these services; as the project unfolded there emerged differing expectations about the need to reach large numbers rather than just testing models; and a mid-term evaluation discouraged continuation of the model tests. While this area was considered a disappointment, other areas were thought to be more successful, including training of intermediaries (such as providers and parents) in ARH; institutionalization of pre-service training in nursing schools; a mass media campaign; policy development related to access of minors to contraceptives; and the establishment of a National Youth Policy. While Youth.now’s final record was mixed, several of its initiatives have been sustained, and its visible outreach to the public and to professionals has deepened understanding of ARH issues in Jamaica. JA-STYLE, with its expanded portfolio, faced many challenges: There was a delay in putting top staff in place; the MOH continued to resist separate services for adolescents; and, most important, time and funding cuts were announced soon after the staff was in place. Nevertheless, JA-STYLE got a considerable amount done: media exposure; development and dissemination of the IPR; establishment of an active and effective youth advocacy network; a small grants program that helped a number of CBOs to carry out important work (including some that helped others to do so); initiation of collaborative activities in violence-prone communities; and the start of useful capacity-building activities with community groups and NGOs. That many of these activities did not appear to last beyond the funding or be sufficiently integrated and focused to effect change is due mainly to the time and funding cuts but also in part to a lack of strategic planning in the early stages. (The actual impact of project activities will not be known until after the impact analysis.) Even with its brief tenure, however, some activities could be sustainable, among them the IPR (already partially adopted and with certification pending); and inclusion of ARH in their activities by some partners, such as SDC, the 4H), and possibly the J-YAN, should it attract sponsorship. The MOH as a key collaborator on these projects reported mixed results to the team. For example, there were differing views on whether the thematic areas should have been expanded to substance abuse and violence. More representatives appeared to favor this expansion than not, but most were unsure if the issues had been truly addressed. In the case of JA-STYLE, the MOH also reported considerably less collaboration than the project did. Change of government, reorganization of ministries, and especially changes in personnel at key MOH posts affected the potential for achieving project agendas. Several issues need to be sorted out before effective action on ARH or healthy lifestyle agendas can proceed effectively. Among them are the conflict between the CCPA and the policy about availability of contraception to youth; the approach for delivering ARH services; and where ARH should be sited the MOH. All these seem resolvable, especially the ARH services issue, which has become too polarized between the options of separate services for youth and the same services for all. In fact, it would be possible to make some accommodation for young clients short of creating parallel services. USAID/J has been a solid supporter of ARH and healthy lifestyles since 2000. Its funding has enabled models to be tested, important issues to be exposed, and a spotlight to be placed on the importance of 28 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 M&E. It also helped to emphasize the importance of policy and to identify partners that could join in the work to support adolescent healthy lifestyles. Still needed are work with the MOH to sort out unresolved ARH issues and a serious effort to help promote social norm change on sexuality and gender issues. Until this is done, progress will be slow on preventing early pregnancies, multiple partners, HIV and STI transmission, substance abuse, and violence. Although work is still needed on healthy lifestyle issues for youth, funding from USAID/J will be less available to carry it out. Jamaica has “graduated” to a level of RH achievement that brings it to the point where USAID is considering phasing out its family planning/RH support. According to the Phase Out Plan, contraceptive prevalence among 15–19-year-olds has reached 70 percent. The fertility rate (births per 1,000) for the same age group dropped from 122 in 1983 to 79 in 2002. USAID significantly contributed to these achievements; it is now hoped that the initiatives started and institutional changes effected can be sustained to carry forward work in Jamaica to improve adolescent RH and for them to attain healthy lifestyles. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 29 VII. RECOMMENDATIONS JA-STYLE For the rest of the project period, JA-STYLE project should focus on interventions that have the highest probability of wide reach and sustainability: With less than a year to go JA-STYLE should maximize its potential by focusing human and financial resources on those projects that can reach the greatest number in target audiences; foster a positive change in the environment for healthy lifestyle education and service programming; and further build the capacity of organizations that can continue to work on these issues. Therefore, JA-STYLE could continue and expand its mass media efforts; work on social norm and policy change to increase support for ARH and healthy lifestyle issues; and provide TA for improved the sustainability of select NGOs. USAID/J should facilitate a dialogue with the MOH on achievement of a consistent vision and feasible approach to delivery of ARH services: Given the long-standing differences of views about delivery of ARH services as separate or integrated services that have constrained efforts to move forward on ARH, a frank dialogue about options, pros and cons, feasibility, and other matters needs to be initiated. The significant research and experience in Jamaica and other countries can lend insight and ideas for building consensus. Efforts should be made to find an appropriate place for ARH within the MOH: Another matter that has compromised efforts to move forward on ARH in a collegial manner is the practical issue of where ARH activities should be positioned within the MOH. Although during both projects it was sited within the Office of Health Promotion and Protection, many believe that Family Health would have provided a more productive home for it. However, until the matter of how to deliver services to adolescents is resolved, a change in placement would not produce better results for ARH. Options should be explored for achieving an effective intersectoral approach to addressing youth development: In addition to dialogue on delivery of health services to adolescents and the focus of ARH within the MOH, there is a need to better define adolescent health and development as a multisectoral issue, with concerns related to areas like health, education, labor, sports, and security. While it is difficult to achieve an effective working relationship among numerous sectors that have unequal resources and strength, innovation and more determined efforts to try new approaches to strengthen adolescent development programming would be beneficial. Support for community-based condom distribution should be increased: Given that many young people do not use health facilities, getting condoms to them where they live, work, and play is essential. Distribution through pharmacists could be greatly expanded (only five have yet been trained). Other outlets could be grantees, community partners, and small merchants. IPR should be refined and focused to prepare effectively both clinical and nonclinical providers in the approach: The IPR program grew out of a widely identified need. To ensure that further efforts are meaningful and sustained, it is essential that a methodology be laid out that can be easily followed for replication. Given their limited reach, ineffectiveness, and poor connection to services, the adolescent centers should be phased out: There is a substantial literature, with numerous examples, noting the failure of youth centers to link youth to or increase their use of services. If there is a continued commitment to the two sites, there must be evaluations, tighter linkages, better publicity, and significant community mobilization to increase the potential for their success. 30 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 JA-STYLE’s approach should be altered to enhance rather than restrict adolescent access to condoms: JA-STYLE should clearly articulate its support of increased access to condoms for sexually active youth and should encourage all partners and grantees to make condoms available. Establishing an independent nonprofit organization for J-YAN should be supported: J-YAN has demonstrated an entrepreneurial spirit and considerable capacity and initiative in pursuing its—and JA￾STYLE’s—objectives, often with individual members pursuing activities they are committed to or where they have connections to tap into. This is a potentially sustainable group, especially if a parent or sponsor can be identified. ARH issues should be integrated into media messages and TV and radio appearances whenever possible: Given that ARH is an integral part of healthy lifestyles and a key theme in JA-STYLE’s mandate, integration should be pursued, especially where it does not require additional expenditure of resources. This will help to increase the understanding of where ARH fits into the larger picture. Discussion guides and vignettes for “Outta Road” and “Curfew” should be distributed to facilitate their use in a variety of settings: Several informants encouraged the expansion of “Outta Road” and “Curfew” materials to extend their shelf life and enable more varied and interactive use in a greater variety of places, such as educational institutions and community groups. Efforts with organizations still receiving grants should be focused on building their capacity and sustainability: To increase the chances of sustainable ARH actions over the long term, it is important for JA-STYLE to ramp up capacity building with the organizations it still supports. These efforts should be based on careful examination of each organization’s unique needs (utilizing the OAT, as appropriate), and TA should be provided to address the gaps identified. The end-of-project meeting should identify completed and pending activities and products for possible future use: The end-of-project meeting JA-STYLE is already planning with major stakeholders offers a useful opportunity to review results, share lessons learned, and identify actions to be taken and activities to be sustained by other entities. Identifying and sharing completed and pending activities and products (such as training curricula, job aids, PSAs, and “Outta Road” radio programs) should be emphasized, so that the broader community can benefit from JA-STYLE’s outputs after the project has ended. This is particularly important given the limited carryover between Youth.now and JA-STYLE activities. There should be a plan for continuing an organization only if an effective, cohesive operational entity could advance the ARH and youth development agenda: Although JA-STYLE has contributed to supporting adolescent healthy lifestyles in Jamaica over the past years, it does not seem feasible or strategic to continue the project as it currently exists. In short, JA-STYLE’s lack of cohesive focus and strategic niche does not lend itself to its being carried over as is. HIV/AIDS Listed below are the evaluation team’s best recommendations for relating future HIV/AIDS funds to youth programming. The evaluation team recognizes that implementing the recommendations is heavily dependent on the legal policy environment in Jamaica, USAID funding policy, and the availability of resources at the time. • Ensure that the new MOE-developed Health and Family Life Education (HFLE) curriculum for grades 7–9 is implemented as envisioned with regard to condom discussions and demonstrations. The MOH and other stakeholders need to be vigilant to ensure that implementation does not depend on the comfort level or influence of persons delivering the content, especially given the ambivalence among teachers, other school personnel, and the public about condom discussions, demonstrations, and availability to and use by minors. Execution of the curriculum as intended Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 31 should naturally lead to discussion in the community, resulting in improved access through policies that allow condom distribution in schools. • Explore promotion of policies that would make condoms available in schools. • Ensure access to HFLE for out-of-school youth, those in correctional facilities, and other vulnerable groups. The MOH’s HFLE curriculum for grades 7–9 covers ARH topics, but exposure to it is limited to in-school youth. Its application should be expanded to more vulnerable youth. • Take action to teach males about condom use at earlier ages. The mean age of first sex for males in Jamaica is very low, which implies an increased number of lifetime partners. These realities suggest that encouraging abstinence for the very young should be coupled with reaching them with information and education about how to use condoms. Condom use can then be normalized at an early age, coincide with first sex, and help prevent pregnancy and the transmission of STIs, especially HIV/AIDS. • Scale up HIV testing, including VCT, targeting the most vulnerable populations, such as MSMs, CSWs, and disabled youth. The drive to scale up HIV testing should be continued, with VCT being offered to all persons who undergo testing. Though VCT only for persons who test positive is more common, since some studies have shown increased risk after a negative test result, counseling to explore how to reduce personal risk and promote behavioral change should be heavily promoted even for persons who test negative. Emphasizing vulnerable populations will help keep them HIV-negative and reduce transmission rates. • Implement a public education campaign to address social norms on sexuality and gender issues, such as sexual risk-taking, sexual abuse, availability and acceptability of condoms, and linkages between violence, substance abuse, and HIV. Several aspects of the Jamaican culture have influenced the increased transmission of HIV. Those related to gender and sexuality issues have been difficult to discuss publicly; keeping these subjects taboo will only delay the necessary recognition of factors fostering early pregnancy and HIV transmission. Thus, a public education campaign is needed to address social norms specific to sexuality and gender roles. The campaign should speak to issues that seem engraved in the culture, such as the current acceptance of multiple partners and domestic violence, inconsistent condom use, early sexual initiation, homophobia, and forced and coerced sex. Such a campaign also needs to explore thoroughly the links among many of these issues (violence, both physical and sexual; substance abuse; and HIV). • Support J-YAN to carry out youth-driven activities related to HIV prevention that are targeted to the most marginalized groupsJ-YAN’s large membership illustrates that it has the capacity and the expertise to reach the most marginalized youth groups island-wide. Its leaders have expressed an interest in making it an independent NGO and have apparently received an expression of interest from a potential parent organization. J-YAN should be supported and further capacitated to carry out youth-driven HIV prevention activities. 32 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 33 APPENDIX A. SCOPE OF WORK [NOTE: This document was amended by agreement between the evaluation team, USAID/J, and GH Tech. The major changes that were made relate to priority tasks, team composition, and assignments by team members. Changes were also made in level of effort and timeline (not reflected in the text). EVALUATION OF THE JAMAICA ADOLESCENT REPRODUCTIVE HEALTH ACTIVITY (2000–2006) (Revised GH Tech: September 28, 2007) STRATEGY PERIOD: (2000–2004) Project Name: The Adolescent Reproductive Health Project Project Number: 532-0184 Authorized LOP Funding: US$12.0 million PACD: September 30, 2004 STRATEGY PERIOD: 2005–2009 Project Name: The Jamaica Adolescent Healthy Lifestyle Project Project Number: 532- 011 Authorized LOP Funding: US$14.9 million Award: January 7, 2005 PACD: September 30, 2009 Amendment to Contract: March 28, 2007 Subcontractors: Advocates for Youth Health Strategies International Population Media Center ASHE Caribbean Performing Arts Foundation BACKGROUND Adolescent reproductive health is a critical area of concern for Jamaicans under the age of 20, who constitute 20 percent of the population. Total fertility rates have declined significantly throughout the island since the 1960s. The rate among females 15–19 years old is now at 79 births per thousand population. The Adolescent Reproductive Health Project was initiated in 2000 to address the adolescent reproductive health (ARH) needs of Jamaican youths. Some of the main untoward outcomes and risky behavior associated with adolescents’ lack of preparation for healthy reproductive behavior include interrupted and discontinued schooling, a high incidence of pregnancy and related complications and of HIV/AIDS among youth, and unacceptable levels of drug abuse among adolescents coupled with 34 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 spiraling crime and violence. The situation is further exacerbated by the lack of social and economic preparation for effective parenting. Since 2000, USAID/J-CAR has placed the lion’s share of its health budget on improving the reproductive health of adolescents. During the five-year strategy (2000–2004) the Mission’s strategic objective for health was aimed at improving the RH of youth by increasing the use of quality RH and HIV/STI services and preventive practices. This five-year US$12.0 million activity was designed around the premise that increased use of RH services and participation in youth-friendly RH activities can be achieved by: • increasing access to quality RH and HIV/STI services; • improving knowledge and skills related to RH and HIV/AIDS/STIs; and • supporting RH national policies and guidelines. The Futures Group International (TFGI), on behalf of the Ministry of Health (MOH), was awarded the contract for the strategy period 2000–04 to implement ARH activities. Additional activities for adolescents were implemented by a diverse group of international technical assistance firms, local nongovernmental organizations (NGOs), and other ministries. TFGI’s work included offering a comprehensive package of technical assistance, training, and upgrading supplies and equipment to improve the quality and accessibility of services to youth. In addition, TFGI developed innovative approaches to providing reproductive messages to youth, monitored behavior change and strengthened nursing and medical curricula as they relate to the RH of youth and played a supportive role in drafting national policies and guidelines. Work being carried out by other organizations included strategic planning and policy formulation, quality assurance, school-based ARH interventions, research studies and behavioral surveillance surveys, and training. Key ARH outputs developed during the strategy period (2000–2004) included establishment of youth friendly sites, development and implementation of a comprehensive mass media campaign, implementation of a training program, production of an ARH learning package, establishment of a small grants program, and initiation of a public relations and advocacy component. Strategic planning RH workshops on the multisectoral approach to youth development were also completed, resulting in a National Youth Policy. Quality assurance interventions on MCH and family planning standards were implemented and youth-friendly standards and criteria developed. The Vibes curriculum, which was designed and developed with the Ministry of Education (MOE), was disseminated. ARH surveys were also designed, developed, and disseminated, building on the evidence-based data that are available on ARH in Jamaica. In 2005, USAID/J-CAR awarded a contract to University Research Co., LLC (URC) (URC) to improve the health status among vulnerable groups (10–19 years). The contract is based on the concept of increasing healthy behaviors among adolescents by: • Expanding access to youth-friendly services in clinical and nonclinical settings to promote healthy lifestyles and appropriate sexual behavior; • Implementing national policies and guidelines in support of healthy lifestyles (with a focus on youth sexual practices); • Improving knowledge, attitudes, and skills related to healthy lifestyles and appropriate sexual behavior; and Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 35 • Increasing community support and involvement in promoting appropriate sexual behavior among adolescents. The major expected program outcomes are planned to build on successes of the previous strategy, strengthen the capacity of the MOH to reduce high-risk sexual behavior among the adolescent populace, strengthen the capacity of NGOs to be self-sustaining in delivery of healthy lifestyle activities, and increase the number of youth-relevant educational messages. The interventions are also intended to increase the involvement and improve the capacity of the community to take responsibility for promoting healthy lifestyles among adolescents. The interventions support the implementation of policies related to RH rights and issues within the ambit of healthy lifestyles. The interventions are intended to be multifaceted. They constitute a comprehensive package of activities designed to reduce sexual risk-taking by adolescents and bring about a reduction in unintended pregnancies, HIV/AIDS, and drug abuse and violence. A youth resiliency model that attempts to enhance protective factors is included as an overarching concept. In addition, planned interventions are to adopt an intersectoral/multisectoral approach, building on existing institutions that implement ARH activities with a view to the sustainability of program activities. The URC project was to have a duration of five years. Mission funding cuts have led to a significant reduction of project funding and a shortening of the project. Two years and eight months into implementation of its activities URC‘s accomplishments include • Roll out of the Interpersonal Relations Learning program and supporting communications materials and job aids to regional health authorities island–wide • Development of a package for counseling adolescents • A literature review of approaches to triaging and managing adolescents seeking sexual and RH services, including voluntary counseling and testing (VCT) • Establishment of two youth-friendly sites • Reinstatement of an adolescent policy working group in the MOH • Partnerships sustained with the National Family Planning Board to train service providers on the guidelines for providing contraceptives to sexually active minors • Expansion of youth advocacy networks expanded and design and dissemination of a youth advocacy toolkit and training manual • Provision of technical support to the National Youth Policy and a national parenting policy • Public/private partnerships initiated through Mentor, FIFA, and the National Endowment Fund at the University of the West Indies • Created and aired a radio serial drama with sponsorship from Digicel (45 episodes aired) • Supported the development of the theatrical performance and educational DVD of the violence prevention musical Curfew in collaboration with ASHE Caribbean Performing Arts Foundation • Trained young people and local graphic artists in innovative photo-to-graphic methodology • Trained caregivers and parents in ARH and trained grantees in behavior change communication (BCC) • Supported Healthy Lifestyle Clubs in schools in collaboration with the MOH 36 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 • Contributed to the development of a parenting skills curriculum and a “Good Parenting” calendar • Supported youth development organizations operating at a national level • Initiated community interventions in violence-prone areas • Provided support to grantees in seven parishes to implement adolescent RH and parenting skills activities • Developed a curriculum on adolescent healthy lifestyles issues and trained SDC outreach workers • Drafted a sustainability plan. Since USAID began, its support to ARH in Jamaica in 2000 it has monitored results by collecting both quantitative and qualitative data from multiple sources. Sources of primary quantitative data include community sample surveys (baseline and endline) collected in target and control communities; exit interviews and mystery client studies; post-launch media communication surveys; policy environment assessment score (PEAS) survey; and simple service statistics and information monitoring systems at the community and parish levels. Qualitative analyses include key informant interviews conducted in select communities. The project management system (PMIS) has data collection tools and instructions, data entry worksheets, and output report spreadsheets and templates for indicators. This evaluation will examine the impact of USAID support to ARH in light of the interventions that have been developed and their relation to sustainability and institutionalization of activities. The results of the evaluation will be used to inform and share lessons learned about the impact of USAID support to ARH in Jamaica with various stakeholders and the wider international community. In addition, the results will provide information to governmental bodies and private institutions with a view to leveraging the additional funds necessary for the institutionalization and sustainability of program activities and accomplishments. PURPOSE The overall purpose of this evaluation is threefold. The first is to assess the impact since 2000 of USAID/J-CAR’s contribution to improving the RH outcomes of adolescents in Jamaica. The second is to assess the progress and achievements of the USAID JA-STYLE project in terms of project design, implementation, and positioning of the project for institutionalization and sustainability of its program interventions. The evaluation will examine the progress, impact, application, and replicability of major project inputs within the health, education, and social sectors. The third is to assist USAID/Jamaica with its strategic design and focus for the next strategy period. As Jamaica graduates from receiving family planning/ RH funding from USAID, the budget for the adolescent RH program will be drastically curtailed. However, the mission anticipates that it will continue to receive a modest amount of HIV/AIDS funds in the future. The specific objectives of the evaluation are to • review USAID/J’s strategy and approach to adolescent reproductive health (ARH) in Jamaica since 2000 within the context and scope of the support to ARH received from other sources, such as the external donor community and sectoral ministries; • analyze the impact of USAID/J’s contribution to improving the RH outcomes of adolescents in Jamaica, which will require both quantitative and qualitative analyses—the analysis will not rely exclusively on face-to-face discussions, consultations and interviews but will also utilize population trend analyses, surveys at the community level, and secondary data sources; Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 37 • assess the extent to which the institutional environment in the health, education, and social sectors and the NGO community are conducive to the institutionalization and sustainability of ARH interventions—factors to be addressed include current policies and practices, financial capability, physical infrastructure, and political will; • provide USAID/J with a comprehensive report on the impact, outcome, and effectiveness of their support to ARH in Jamaica, along with performance on the contract to date, with recommendations for the implementation of activities for the remaining life of the project; and • provide USAID/J with strategic direction that will assist the mission in making critical decisions regarding the new health strategy (because Jamaica is among countries that are being phased out from receiving population/family planning funding, the mission anticipates that only HIV funding will be available for future programming). STATEMENT OF WORK (SOW) The evaluation team will be responsible for carrying out a comprehensive review of USAID/J support to ARH since 2000 as well as URC program planning, administrative, management, and implementation of the contract since inception. The team will also provide strategic direction to USAID/J with respect to the interventions that are best suited for its health program for the next strategy period. The team should rely where feasible on the activity results framework, associated results statements, and related indicators and targets. The parameters for this effort are described below. A. Assess USAID/Jamaica’s ARH interventions. USAID/Jamaica commenced its support to ARH in 2000 (the contract was signed in November 1999). The institutional contractor (The Futures Group International) and other cooperating agencies (CAs) implemented a diverse array of activities. In February 2004, USAID awarded University Research CO., LLC (URC) a contract as a follow-on to the Futures contract. The evaluation will determine what has been accomplished to date for the various interventions, their effect vis-à-vis stated objectives, potential for replication, institutionalization, and sustainability of inputs. The primary focus will be on the sustained impact of the interventions performed by the Futures Group (Youth.now) and the effect of the activities implemented by URC. (JA-STYLE). Other supportive CA contributions are much smaller and of a discrete, time-limited nature and not designed to be self-sustaining. This will be taken into account in the evaluation process. Evaluation question: What is the impact of USAID support to ARH since 2000? Since 2000, USAID/J has provided financial resources in the amount of US$15 million to Jamaica to improve the reproductive health of adolescents. As the Jamaican Family Planning Program faces phase￾out of population assistance, USAID/J seeks to assess progress made toward improving adolescent RH outcomes at both a macro and a micro level. The analysis will assess the extent to which the planned results are met, the validity of the underlying assumptions, how well the needs of different customers are met, any unintended consequences that emerged, the sustainability of activities, and the effectiveness of the strategy. Evaluation question: How effective are the activities implemented by the current project? The evaluation team will analyze the impact within the construct of the intermediate results: expanding access to youth-friendly services; improving the knowledge and skills of providers of care; increasing community support, and supporting national policies and guidelines. The evaluation team will utilize available data sources, including the Government as well as the project’s MIS system, annual reports, and data information. They will conduct interviews with project personnel, stakeholders, and key personnel in the MOH, MOE, community-based NGOs, and the private sector. They will assess the impact of 38 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 interventions geared toward developing youth-friendly services; the effectiveness of the various mass communication interventions, including the radio serial, in terms of changing attitudes and behaviors; the impact of training efforts; inroads made toward increasing community support; and the program’s support for an improved policy environment for adolescent RH. Importantly, the team will conduct focus group discussions with adolescents to garner information on changes in knowledge, attitude, behavior, and practices with respect to RH issues. Evaluation question: How effective is the URC operation in the overall management of the ARH activities? The major emphasis will be on the robustness of the approach developed, which included expanding access to youth-friendly services, supporting national policies and guidelines that focus on youth sexual behaviors, improving knowledge and skills of providers of care, and increasing community support and involvement in promoting appropriate sexual behavior among adolescents. For example, the evaluation team will determine progress in terms of the project’s contribution to strengthening the capacity of the government and NGOs to improve the interpersonal and counseling skills of providers of care to interact more effectively with adolescents, the impact of the mass communication strategies on behavior change, community involvement and mobilization efforts, and supporting national policies and guidelines to promote a healthy adolescent lifestyle. B. Provide recommendations as to how current ARH activities can best be integrated into organizations for sustainability. USAID/J is already focused on sustaining the impact of its ARH activities. URC and USAID/J, together with the MOH, are currently working to identify ARH activities that can be integrated into other institutions, including the various sectoral ministries and NGOs dealing with youth. We anticipate that a group of key stakeholders will meet in July 2008 to discuss exactly which activities should be integrated and identify future “homes” for these activities. The July meeting will inform smaller work planning meetings with key stakeholders to confirm and clarify work planning objectives. The team should meet with the organizations identified through this process and make recommendations as to how the activities can best be integrated; what additional resources, including funding and/or training/capacity building might be necessary for full-integration; and appropriate timelines. This will be accomplished through consultations, focus group discussions, reviewing reports, and conducting interviews at various levels. C. Recommend follow-on activities to be carried out by the MOH and other stakeholders. Pursuant to activity B above, the team should present specific recommendations for the MOH and other stakeholders that will be assuming responsibility for ARH activities from URC. These recommendations should identify strategies for supporting the activities, as well as gaps that will require leveraging support to assure sustainability over the long term. D. Recommend the way forward with respect to using HIV/AIDS funds to support youth and ARH. As indicated above, USAID/J expects to receive only HIV/AIDS funds in its health portfolio starting with FY 08. Youth comprise Jamaica’s largest vulnerable group. The team should recommend how HIV/AIDS funding can best be used to reach youth with information, and any subsequent recommendations will also be used to help USAID/J update its Strategic Plan for FY2005–2009. It will be an important tool to help USAD/J to make critical decisions related to future programming and to leverage additional funds to sustain the gains made in improving ARH in Jamaica. Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 39 TEAM COMPOSITION AND PARTICIPATION The evaluation will be conducted over a (TBD) period by a multi-disciplinary 6-person team. The evaluation team should consist of: 1) Team Leader/Evaluation Specialist, 2) Adolescent Services Specialist and Policy Advisor, 3) Community Participation Specialist, 4) Two Youth Consultants, 5) Local Liaison; and 6) Social Anthropologist Advisor. The Team Leader/Evaluation Specialist will be responsible for coordinating the activities of the evaluation team. The team leader will provide management support and oversee the development of the evaluation schedule and approach and the findings of different team members and coordinate the preparation of the final reports. The leader will also have responsibility for drafting, including tables, graphs and other quantitative data that will be included in the report. The leader will also ensure timely completion of the evaluation report, including recommendations for follow-on activities. In addition the team leader will provide expert technical advice and advice on evaluation methodology to the evaluation team, including the analysis and interpretation of data required to adequately assess project direction and impact to date. This will require approximately (TBD). Qualifications: MPH or equivalent with emphasis on health research/statistics and extensive experience in evaluating health programs. S(he) will have at least ten years experience working with sexual and RH projects and be proficient in the use of Power Point, Excel, and Word Perfect. This will be approximately (TBD) Experience: At least 10 years of administration of multifaceted adolescent RH projects in developing countries and the Caribbean, preferably in Jamaica. The team leader will have experience in managing large multidisciplinary teams, and the ability to conceptualize and write clearly and concisely. The Adolescent Services Specialist and Policy Advisor will be responsible for assessing the suitability and impact of the interventions developed to increase adolescent access to youth-friendly services in clinical and nonclinical settings. (S)he will examine the long-term effect of project inputs aimed at increasing adolescent access to services in both clinical and nonclinical settings. The individual will examine provider perceptions, approaches, and attitude to adolescents who access services at their facility, and review data available at the various settings to ascertain the trend in utilizing services and the quality of services provided to adolescents. The potential impact of the interpersonal communications experiential learning curriculum for sustainability will also be assessed. In tandem the specialist will analyze the policy environment for adolescents in Jamaica. The individual will also review and analyze polices directed at adolescents and the results of implementation of these policies in various settings. Qualifications: At least 10 years of experience in developing countries including the Caribbean and familiarity with the implementation of ARH services and adolescent policies. An MPH or equivalent in health, social science, or a related area is needed. The Community Participation Specialist will be responsible for assessing the appropriateness and effectiveness of the community interventions developed through the project. (S)he will examine congruence between the project inputs developed or supplied through the project at the community level and through the mass media and the Jamaican health system and will assess the functions carried out by service providers and the sustained multiplier effect in light of their training through the ARH project. This individual will also assess the demand for services at the community level based on the methodologies adopted by the project interventions. Qualifications: At least 10 years of experience in developing countries including the Caribbean and familiarity with the implementation of community based interventions. An MPH or equivalent in health, social science, or a related area is needed. This will be approximately (TBD) 40 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 The two Youth Consultants will provide technical assistance for the evaluation. On an ongoing basis they will provide advice and information to the team that reflects the perspectives of the beneficiary group and the realities of adolescent lifestyle and behavior in Jamaica. The Local Liaison Manager will provide logistical support for the evaluation. S(he) will work with the Chief of Party and team members in preparing for field visits, meetings, and consultations in Jamaica and support the team substantively in management, administration, and documentation of the evaluation. The Social Anthropologist will provide an anthropological/sociological/public health perspective as an interim advisor to the team and share understanding of some of the deep-rooted sociocultural paradigms that are integral to understanding the behavioral innuendoes associated with the risky behaviors of adolescents in Jamaica. Technical Direction The Director, USAID Office of General Development (or her designee), will serve as the Evaluations Cognizant Technical Officer for the evaluation and shall provide formal technical direction and general guidance during this effort. USAID/J-CAR will provide guidance on site visits to selected youth-friendly sites, meetings with key partners from the MOH, other sectoral ministries, and other institutions as appropriate. Jennifer Knight-Johnson Project Management Specialist (Health) USAID/Jamaica Phone (876) 92-63645 Fax (876) 92-99944 Timeline and Level of Effort USAID/J anticipates that the entire evaluation will be completed within four weeks. This includes preparation days, in-country work, and travel to select sites as well as drafting the evaluation. If possible, the assessment should take place in September 2007. A six-day work week is authorized in-country. Level of Effort The estimated level of effort (LOE) for this assessment will be as follows (suggested): Document review/preparation (out of country) 3 days Travel days 2 days Team planning meeting (TPM) (in country) 2 days Present evaluation framework Interviews; meetings + field work 10 days Discussions, final interviews, and wrap–up 2 days Draft report (in country) 5 days Revisions/final report (out of country) 6 days (TL); 2 day (others) Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 41 Total LOE for each team member is estimated as follows: Team leader/ Evaluation Specialist 30 person days Adolescent Services/Policy Advisor 26 person days Community Participation Specialist 26 person days Youth Consultant (x 2 persons) 48 person days Local Liaison Manager 24 person days Social Anthropologist 5 person days Total LOE: 159 person days (estimated) Logistics This evaluation will be carried out by the GHTECH Project. The contractor should provide all other logistical arrangements, such as in-country travel, airport pick-up, and lodging. The USAID/JA-STYLE Project CoP will work with the local Liaison Manager and evaluation team providing inputs into coordination of field visit arrangements. However, the evaluation team will play a significant role in selecting sites. As needed the CoP will arrange an appropriate escort for the team during field trips. USAID/JA-STYLE will also work with the Liaison Manager and the evaluation team to provide administrative support in arranging for meetings and transport where practical. USAID/J will provide key documents and materials for reading and help arrange the in-briefing and debriefing. The USAID CTO for the ARH activity will participate in the evaluation as much as possible. Exact participation will be determined during the TPM but someone from USAID will participate in key meetings with the MOH and other stakeholders and at least some field visits. Report/Deliverables 1. TPM: The team will conduct a two-day team planning meeting (TPM) upon arrival in Jamaica and before starting the in-country portion of the assessment. The TPM will review and clarify any questions on the assessment SOW, draft an initial work plan, develop a data collection plan, finalize the assessment questions, develop the assessment report table of contents, clarify team member roles, and assign drafting responsibilities for the assessment report. The TPM outcomes will be shared with USAID/J and the health team will participate in sections of the TPM. A Local Liaison Officer will be hired to assist with logistical preparations. 2. Initial Briefing: The consultant team will conduct an initial briefing with members from the Office of General Development to discuss the SOW and Plan of Action for the evaluation, and present USAID with the framework for the evaluation on day three of the assessment. This will include the materials produced during the TPM. 3. Draft Report: The first draft of the final assessment report will be due at the end of the team’s visit. The length of the report body text should not exceed 40 pages (format: A4, font type: Times New Roman, font size 12). The draft report will include key findings and recommendations for mission review. A draft report with recommendations for design activity will be submitted to USAID/J-CAR (TBD). 4. Debriefings: The team will conduct one mid-term and one final debriefing. The mid-term debriefing will take place after the field visits to discuss preliminary findings with USAID. The first final debriefing will be with USAID, key representatives from various sectors, and the donor community. The debriefing should present key findings and recommendations in a PowerPoint format. 42 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 5. Final Report: USAID/J-CAR will provide the team with comments on the draft report within 10 days of receiving it. The final report will be due within 10 days after the team receives comments from USAID/J-Car. The report will be finalized and five copies (along with a diskette in [Office XP Microsoft Word Format XP]) will be submitted (TBD). It should be no more than (TBD) pages long. USAID/J-Car requests both an electronic vision of the final report (Microsoft Word or PDF format) and four hard copies of the report. In accordance with the requirements of the SOW, the evaluation contractor will provide the following deliverables: • An evaluation report which will include: o Findings on the impact of implementation and effectiveness to o Quantitative and qualitative data that shows the trend in various elements of ARH since 2000; o Assessment of the impact of program on the various line Ministries that implement programs on behalf of adolescents and the impact of the activities on the beneficiaries. Recommendations for institutionalization and sustainability and Recommendations on the strategic/direction for the new health strategy • The evaluation report should include the following sections: o Executive summary o Introduction o Background of the problem o USAID's assistance to Adolescent Reproductive Health since 2000 o Findings o Conclusions/lessons learned o Recommendations o Annexes (persons contacted, sites visited, and documents consulted) Estimated Budget To be developed upon finalization of SOW and when team members are identified. Drafted: SO11: JKnight-Johnson_______________SOW mid-term eval 2007 ARH activities since 2000.doc Revised: GH Tech 05-10-07 BS; JMK Revised USAID/J 6/5/07 JKnight-Johnson; Revised GHTech 06-08-07 : Revised USAID/J20/6/07; GH Tech 09-28-07 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 43 APPENDIX B. PERSONS CONTACTED Key Informant List Arranged by Organization USAID Bob Burkeness Director, Office of Program Development and Management Leigh Shamblin Director, Office of General Development Jennifer Knight-Johnson Project Management Specialist (Health) Adolescent Healthy Lifestyles Project (CTO) MINISTRY OF HEALTH National Level Sheila Campbell-Forrester Chief Medical Officer of Health Peter Figueroa Chief, Epidemiology and AIDS Lovette Byfield BCC Specialist, National HIV Prevention Programme Joan Clarke Director, Standard Research Standards and Regulation Joi Chambers Project Manager, Adolescent Reproductive Health Health Promotion and Protection Division Takese Foga Director, Health Promotion and Education Regional Level Alex Konstantinov Acting Regional Director, Western Regional Health Authority Beverly Wright Acting Regional Technical Director, Western Regional Health Authority Diahann Dale Regional STI/HIV/AIDS Coordinator, Western Regional Health Authority 44 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 ADOLESCENT CENTRE (AT MANCHESTER HEALTH CENTRE) Faith Davy-Lyttle Coordinator, Adolescent Centre Shereen Williamson Reid Community Peer Educator Terry-Joy Stephenson Targeted Intervention Officer Manchester Health Centre Focus Group Interview with 9 youth (ranging from 12–16 years) KINGSTON SCHOOL OF NURSING Marcia Hyman McKay Director NATIONAL COUNCIL ON DRUG ABUSE Michael Tucker Executive Director Sislyn Malcolm Director, Field Services Ellen Campbell Grizzle Director, Information and Research MINISTRY OF EDUCATION, YOUTH, AND CULTURE Guidance and Counselling Unit Monica Holness Assistant Chief Education Officer Eugenie Brown Prevention Education Programme Coordinator Christopher Graham National Coordinator, HIV and AIDS Education NATIONAL CENTRE FOR YOUTH DEVELOPMENT Ohene Blake Executive Director MINISTRY OF NATIONAL SECURITY Oral Khan Executive Director, Office of the Cabinet National Security Strategy Implementation Unit Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 45 SOCIAL DEVELOPMENT COMMISSION Daniel Wilson Director of Finance and Administration Courtney Brown Regional Director (Region 1) Ishiwawa Hope Regional Community Development Planning & Programmes Manager (Region1) Venta Longman Director of Training VIOLENCE PREVENTION ALLIANCE Deanna Ashley UWI Research Fellow Honorary Research Fellow Elizabeth Ward Co-Chair NATIONAL FAMILY PLANNING BOARD Olivia McDonald Executive Director RED CROSS Lois Hue Deputy Director General CHILDREN FIRST Francine McDonald Youth Wellness Activity Stacey Lacey Assistant Administrator Rhonda Morrison Counselor BASHY BUS EDUTAINMENT PRESENTERS KIDS CAMP Ann Marie Richards Principal, Programme Coordinator 46 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 4H Francis Robinson Parish Development Officer Interview with 10 youth (4H members) FLANKER PEACE AND JUSTICE AND CENTRE Marilyn McIntosh-Nash Director ROTARY CLUB Shirley Platt Club Services Director ANNOTTO BAY HEALTH AND ENVIRONMENT ASSOCIATION Aldon Smikle Chairman WESTMORELAND ASSOCIATION OF CLUBS Mathias Brown Facilitator Interview with 18 Club members (youth, parents, other adults) FAMPLAN Peggy Scott Chief Executive Officer Pauline Pennant Programme Administrator Theresa Gaynor Assistant to CEO WOMEN’S CENTRE (ST. ANN’S BAY) Velma Monteith Centre Manager YMCA Sarah Newland-Martin Director PORTLAND YOUTH CENTRE Kiesha McFarland Intern Interview with 2 youth at Centre Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 47 NATIONAL GIRLS BRIGADE Blossom Hoad Organizer/Training Officer Catherine Lyttle Project Coordinator HOPE WELLNESS CENTRE Kim Scott-Fisher Child Resiliency Programme Director PHARMACIES Noreen Willis Pharmacist—Three Mile Pharmacy Jillian Hendricks Pharmacist—Moodies Pharmacy Howard Richards Pharmacist—New Highgate Pharmacy YOUTH.NOW Pauline Russell-Brown Chief of Party LISTENING GROUP (RUSEAS HIGH SCHOOL, HANOVER) Vedalyn Bowlin Guidance Counsellor, Ruseas High School Interview with 8 students POSITIVE YOUTH IN ACTION (VIOLENCE PRONE COMMUNITY, BROWNS TOWN) Carol Cooke Managing Director Beresford Francis President 48 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 COMMUNITY REPRESENTATIVES/VOLUNTEERS Daren Williams Petrona Barnette Angela Graham Wayne Miller Sachella Grant Whitclif Roden JA-STYLE STAFF Ann Marie Campbell Chief of Party Julie Urban Jaser Project Manager Audrey Crosdale BCC & Service Delivery Specialist Ingrid Thame Monitoring and Evaluation Specialist Ian McKnight Outreach and Communication Specialist Kandasi Levermore NGO, CB, & Community Mobilization Specialist Layne Robinson Youth Policy and Advocacy Specialist Caryl Grant Western Regional Coordinator George Young Consultant &Trainer, Violence Prevention, Youth with Disabilities Sherrian Gray Violence Prevention Specialist Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 49 JA-STYLE YOUTH GROUPS Youth Advisory Board (members ) Vanessa Foote Yannick Hemmings Simone Holness George Newman Jamaica Youth Advocacy Network (members) Andrew Francis Keisha McFarlane Romel Gordon Ezekiel Russel URC Tisna Veldhuyzen van Zanten International Division Director ADVOCATES FOR YOUTH Nicole Cheetham International Division Director POPULATION MEDIA CENTER Kriss Barker Vice President for International Programs OTHER DONORS UNFPA Melissa McNeil-Barrett National Project Officer UNICEF Novia Condell Children and HIV/AIDS Specialist UNAIDS Miriam Maluwa Country Coordinator for Jamaica, the Bahamas & Cuba 50 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 51 APPENDIX C. REFERENCES Ashe Productions, Curfew the Musical, a Michael Holgate Film produced in cooperation with JA-STYLE (DVD) (Kingston: Ashe Productions, 2007). Carr, Peter R., “National Strategic Plan for the Promotion of Healthy Lifestyles in Jamaica 2004–2008,” prepared for Ministry of Health (undated). Center for Disease Control, “Highlights from the Jamaica Reproductive Health Survey, 2002–2003” (Atlanta: CDC, 2003), available at: www.cdc.gov/reproductivehealth/Surveys/Jamaica.htm. Accessed November 2007. Evans, Sheila, “Mandeville Clinic: Adolescent-Friendly Assessment” (2007). Farrell, M., J. Knight-Johnson, M. Sancho-Morris, L Schaefer, H. Smith, and L. Stewart, “Phase-Out Plan for USAID Assistance in Family Planning/Reproductive Health, Jamaica” (2006). Figueroa, J. Peter, “Prevention Review 07,” PowerPoint presentation at the Annual HIV/STI Control Programme Retreat, October 17, 2007, Ocho Rios, Jamaica. The Futures Group International, “End of Project Report. Jamaica Adolescent Reproductive Health Activity” (Kingston: TFGI, 2005). __________, Reproductive Health in Jamaica: Volume I, Analysis of Current Reproductive Health Status, Gaps, Needs, and Opportunities, prepared for the Ministry of Health (Washington: TFGI/The POLICY Project, 1999). __________, “Youth.now: Strategic Training Plan,” prepared for USAID Jamaica Mission (Kingston: TFGI, 2002). __________, Youth.now Project, and National Christian Youth Coalition. “Youth.now and the Church. Individual Choice or Community Responsibility: The Community of Faith’s Response to Adolescent Sexuality—Report, Conference of Religious Leaders” (Kingston: Youth.now, 2003). Gayle, Herbert. “Westmoreland Adolescent Reproductive Health: An Evaluation of Youth.now’s Peer Educator/ Peer Link Intervention” (Kingston: Youth.now, 2004. Harris, Sonia, and Douglas McIntosh, Advocacy Tool Kit for Advocates in Adolescent Sexual and Reproductive Health (Kingston: Youth.now, 2003). 52 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Hope Enterprises Ltd., “Report of Adolescent Condom Survey, Jamaica 2001, Commercial Market Strategies Project (USAID)” (Kingston: Hope Enterprises, Ltd., 2001. __________, “Youth.now Advertising Recall Survey—Final Report,” prepared for Dunlop Corbin Communications Ltd. (Kingston: Hope Enterprises, Ltd., 2004). __________, and Rural Family Support Organization, “What Protects Teenagers from Risk Behaviours? Applying a Resiliency Approach to Adolescent Reproductive Health in Jamaica,” Resiliency Research Report (Kingston: Change Project [USAID], 2001). Jamaica Adolescent Reproductive Health Activity (Youth.now), “Jamaica Guidelines for Adolescent Reproductive Health Service Delivery” (Kingston: Youth.now, 2003). Jamaica Ministry of Health, Life Skills Resource Manual: towards a Healthy Lifestyle, completed by Elaine Chambers (Kingston: Healthy Lifestyles Project, MOH, 2006). __________, “Strategic Framework for Reproductive Health within the Family Health Programme,” final draft (Kingston: Jamaica MOH, 2000). __________, “Summary: Strategic Framework for Reproductive Health within the Family Health Programme 2000–2005” (Kingston: MOH, 2000). JA-STYLE, Adolescent Healthy Lifestyles, training course (DVD) (Kingston: JA-STYLE, 2007). __________, “Adolescent Healthy Lifestyles Project. FY2005 Annual Performance Report” (Kingston: JA-STYLE, 2005). __________, “Adolescent Healthy Lifestyles Project. Life of Project Plan. February 28, 2005–February 27, 2010” (Kingston: JA-STYLE, 2005). __________, “Adolescent Healthy Lifestyles Project. Performance Monitoring Plan” (Kingston: JA￾STYLE, 2005). __________, Advocacy Toolkit (Kingston: JA-STYLE, 2007). __________, “Good Parenting” (Kingston: JA-STYLE, 2007). __________, Highlights. Issues 1–37 (Kingston: JA-STYLE, 2006–2007). __________, “Jamaica Grants Manual” (Kingston: JA-STYLE, 2005). __________, “JA-STYLE FY2006 Annual Performance Report” (Kingston: JA-STYLE, 2006). __________, “JA-STYLE FY2007 Annual Performance Report” (Kingston: JA-STYLE, 2007). Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 53 __________, “JA-STYLE Impact Analysis Plan (IAP): A Conceptual Roadmap,” final draft (Kingston: JA-STYLE, 2007). __________, “JA-STYLE Quarterly Performance Report No. 7” (Kingston: JA-STYLE, 2007). __________, “9-Day Study Tour Johannesburg, South Africa Report,” prepared for review by USAID (Kingston: JA-STYLE, 2006). __________, Outta Road, episodes 1-17, produced for RJR 94 FM Radio (DVD) (Kingston: JA-STYLE, undated). __________, “Sustainability Strategic Plan” (Kingston: JA-STYLE, 2007). National Centre for Youth Development (NCYD), Ministry of Education, Youth & Culture, “National Youth Policy: Jamaican Youth Sharing the World” (Kingston: NCYD, 2004). National Council on Drug Abuse, “NCDA Update,” vol. 9, no 3 (Kingston: NCDA, July-September 2007). __________, “Report, National School Survey 2006 Jamaica,” abridged version (Kingston: NCDA, 2006). __________, “Third National/ Master Drug Abuse Prevention, Treatment and Control Plan (2003- 2008). Five-Year Action Plan for Substance Abuse Prevention, Treatment and Control,” Kingston: NCDA, 2004). Peart, Moses, “Interpersonal Relations Experiential Learning Programme for Enhanced Customer Service in Jamaica’s Health Care Facilities” (Kingston: JA-STYLE, 2006). __________, “Interpersonal Relations Learning Programme for Enhanced Customer Service in Jamaica’s Health Care Facilities,” Facilitators’ Guide (Kingston: JA-STYLE, undated). President’s Emergency Plan for AIDS Relief (PEPFAR), “Guidance to United States Government In￾Country Staff and Implementing Partners Applying the ABC Approach to Preventing Sexually￾Transmitted HIV Infections Within the President’s Emergency Plan for AIDS Relief,” final guidance (Washington: Office of the US Global AIDS Coordinator, PEPFAR, 2005). PSEARCH Associates Ltd. “Sexual Reproductive Health: YMCA’s Response via Youth. now’s Uplifting Adolescents’ Programme (UAP). Near-baseline Evaluation Report,” prepared for Youth.now (Kingston: PSEARCH Associates, Ltd., 2003). 54 Evaluation of the Jamaica Adolescent Reproductive Health Activity 2000–2007 Tiffany, Terrence, Shanti Conly, Everold Hosein, Pansy Hamilton, and Sandy Wilcox, “Mid-Term Evaluation: The USAID/Jamaica Adolescent Reproductive Health Program,” submitted to USAID/Jamaica (Washington: POPTECH Project, 2003). USAID/Haiti, “Award/Contract. The Adolescent Reproductive Health Project,” Regional Contracting Office (RCO) (Port-au-Prince: USAID/Haiti, 1999). USAID/Jamaica, “Amendment of Solicitation/Modification of Contract” (Kingston: USAID/Jamaica Acquisitions Management Unit, 2007). USAID/J-CAR, “Solicitation, Offer and Award. The Jamaica Adolescent Healthy Lifestyle Project” (Kingston: USAID/J Executive Office/Acquisition Management Unit, 2005). “Violence Prevention Alliance – Jamaica Chapter Launched – November 10, 2004,” http://www.who.int/violence_injury_prevention/violence/activities/vpa_jamaica.pdf. Accessed November 2007. Wynter, W., K. Hardee, and P. Russell-Brown, “The Policy Environment Score: Measuring the Degree to Which the Policy Environment in Jamaica Supports Effective Policies and Programmes for Adolescent Reproductive Health,” 2002 Round (Kingston: Jamaica Adolescent Reproductive Health Project [Youth.now], 2003). Youth.now, “Christian Family Life Education: A Resource Guide for Educators and Facilitators. Adolescent Sexuality and Reproductive Health” (Kingston: Youth.now, undated). __________, “Strategic Training Plan,” submitted to USAID (Kingston: Youth.now, 2002). __________, “Work Plan, October 1, 2001 to September 30, 2002 and Life of Project Implementation Plan, October 1, 2000 to September 30, 2004,” submitted to USAID/Jamaica (Kingston: Youth.now, 2001). __________, “YOW Video Series Discussion Guide” (Kingston: Youth.now, undated). For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com