MID-TERM EVALUATION: THE USAID/JAMAICA ADOLESCENT REPRODUCTIVE HEALTH PROGRAM February 12, 2003 Terrence Tiffany Shanti Conly Everold Hosein Pansy Hamilton Sandy Wilcox 2 ACKNOWLEDGEMENTS This report was prepared for USAID/Jamaica by consultants contracted through USAID’s Population Technical Assistance (POPTECH) Project (HRC-00-00-00007-00). The authors wish to express their gratitude to the many public, private and NGO sector participants involved in adolescent reproductive health activities in Jamaica who were contacted by the evaluation team during the preparation of this report. Without their invaluable information, insights and ideas on ARH which they shared so freely with the team, this report would not have been possible. Special thanks go to Jennifer Knight-Johnson, Project Management Specialist and to Margaret Sancho, Director, Office of General Development, USAID/Jamaica for the leadership, support and guidance they provided both for this evaluation as well as for USAID’s continuing efforts to help improve the reproductive health of Jamaica’s youth. 3 TABLE OF CONTENTS Executive Summary …….……………………………………………..………………….. . 4 Abbreviations ………………………………………………………………………………. 6 I. Introduction ……………………………………………………………………………. 8 II. Background …………………………………………………………………………….. 9 III. Program Performance …………………………………………………………... 15 IV. A. Fertility and ContraceptiveUse……………………………………………………. 15 B. Increased Access to Reproductive Health Services………………………………. . 18 C. Improved Knowledge and Skills …………………………………………………. . 27 D. National Policies and Guidelines ………………………………………………… 42 IV. Project Management…………………………………………………………………... 46 V. Program Sustainability ….………………………………………………………….. 47 VI. Conclusion …………………………………………………………………………… 50 Annexes: A. Scope of Work B. List of Contacts C. Results Framework D. Youth.now NGO Subgrants E. Summary Training Chart 4 EXECUTIVE SUMMARY The purpose of this mid-term evaluation is to assess the progress and effectiveness of the USAID Adolescent Reproductive Health (ARH) program in Jamaica from its inception approximately three years ago and to make recommendations for necessary modifications over the remaining life of the program (Annex A). While the Futures Group’s Youth.now project is the primary focus, the evaluation also includes a review of the performance of other related Cooperating Agency (CA) ARH projects and activities funded independently by USAID/Jamaica. The ARH program is designed primarily to address the persistent problem of high fertility and unplanned births among Jamaica’s adolescent population. A number of key indicators of program performance are included in the program’s Results Framework (RF) (Annex C) and in Youth.now’s Results Monitoring and Reporting Plan (RMRP). The RF and the RMRP are the principal standards against which performance is assessed. In carrying out its task, the evaluation team reviewed an extensive number of documents and reports, carried out site visits, and interviewed a large number of key informants ranging from senior policy makers to adolescent peer counselors (Annex B). The team prepared a summary of major findings and recommendations, which was presented and discussed in Kingston at a meeting of USAID/Jamaica, Ministry of Health (MOH) and The Futures Group International (FGI)/Youth.now representatives on November 21, 2002. This evaluation report consists of detailed elaboration and justification of those findings and recommendations. The report begins with a background description of the international and Jamaican experience in ARH, followed by sections describing program performance in the areas of fertility and contraceptive use, increased access to ARH services, improved knowledge and skills related to RH, implementation of national policies and guidelines in support of RH, and project management and sustainability. The key findings and recommendations in each of these areas are: · Fertility and contraceptive use – The program does not appear to be resulting in significant increased contraceptive use among sexually active adolescents; therefore, new approaches are recommended, including greater efforts to reach adolescents through private and commercial sector channels and new efforts to reach older teens as well as new adolescent mothers in clinical settings with hormonal and other methods of contraception. · Increased access to RH services - Youth Friendly Service (YFS) models and sites in MOH clinic settings and in Non-Governmental Organizations (NGOs) designed to reach in-school and out-of-school youth are reaching too few youth to justify the intensive level of effort required; therefore, the YFS sites should be phased out in favor of more broad based training programs for providers and other key groups, since training programs do seem to be having a positive, across-the-board effect on increasing awareness and generating support for ARH programs and activities. · Improved knowledge and skills – Most of the key elements (e.g. a highly competent implementation capacity) are in place for an effective behaviorally-focused communications (BFC) program, though current efforts lack a sharp focus on behavioral outcomes along with 5 BFC strategies linked to those outcomes. Therefore, the Youth.now project needs a sharper behavioral focus, including enhancements in the areas of advertising, branded ARH services, advocacy communication, public relations and information dissemination. Training efforts have been especially successful thus far and should be expanded. · National ARH policies and guidelines – The Policy II Project and Youth.now have made significant contributions to the national ARH policy agenda, though Youth.now currently lacks a strategic focus for both its national policy efforts and related work in policy advocacy; therefore, more program funding and high level support for policy reform is recommended, as well as improved efforts by Youth.now in the area of policy planning and advocacy. · Project management - The Futures Group has performed well in managing the Youth.now project, though its performance would be enhanced with the addition of a MOH “focal point” to help coordinate and facilitate project activities within the Ministry. · Program sustainability – though it is still relatively early in the project, much of the current Youth.now effort does not appear sustainable. That is, many Youth.now interventions are either ineffective, too costly, or because of their breadth and complexity fall outside of the more limited mandates of GOJ and NGO organizations. Moreover, institutional weaknesses and constraints within both the MOH and the NGO sector make adoption and expansion of ARH activities problematic. It is recommended that Youth.now incorporate systems for identifying approaches and interventions that have potential for sustainability along with developing plans to roll-out promising activities in a cost-effective manner. The report concludes that, while it is recognized that much hard work by many skilled and dedicated Youth.now and other professionals have gone into this effort thus far, the results have been less than promising; therefore, it is necessary to take action to refocus the program along the lines recommended in the report. 6 ABBREVIATIONS: ARH Adolescent Reproductive Health ACS Adolescent Condom Survey BCC Behavior Change Communication BCI Behavior Change Intervention BF Behavior Focus BFC Behavior Focused Communication BSS Behavioral Surveillance Survey CA Cooperating Agency CARIMAC Caribbean Institute of Media and Communication CDC Centers for Disease Control CDW Curriculum Development Workshop CMS Commercial Marketing Strategy DCC Dunlop Corbin Communication FHI Family Health International FHS Family Health Services GOJ Government of Jamaica HIV Human Immunodeficiency Virus IR Intermediate Result IEC Information, Education and Communication IUD Intrauterine Device MOH Ministry of Health MSCI Margaret Sanger Center International NFPB National Family Planning Board NGO Non- Governmental Organization NCYD National Center for Youth Development PIOJ Planning Institute of Jamaica PWG Policy Working Group QAP Quality Assurance Program RF Results Framework RH Reproductive Health RMRP Results Monitoring Reporting Plan SERHA South East Regional Health Authority SOAG Strategic Objective Grant Agreement SO Strategic Objective SRHA Southern Regional Health Authority STD Sexually Transmitted Diseases 7 STI Sexually Transmitted Infections TFGI The Future’s Group International TOT Training of Trainers YFS Youth Friendly Services USAID United States Agency for International Development 8 I. INTRODUCTION The overall purpose of this evaluation is to assess the progress and effectiveness of the USAID/Jamaica ARH program since the inception of the Youth.now project in 1999 and to make recommendations in program and project design and implementation for the remaining two years of the current Mission Strategy period. While the emphasis of the evaluation is on the performance of the large Youth.now project, related activities of other Cooperating Agencies (CAs) funded separately by USAID during this period are also addressed. For purposes of this evaluation report, the word “program” is used when referring to the entire USAID-supported ARH effort in Jamaica, which includes both Youth.now and other CA activities. The word “project” is used when referring to Youth.now only. The report is divided up into sections beginning with a Background overview of ARH programs worldwide and in Jamaican. This is followed by a section containing an in-depth assessment of Program Performance, which addresses progress and effectiveness relative to stated objectives and indicators contained in the USAID Strategic Objective #3 Results Framework (Annex C), performance monitoring plans, contracts, work plans and other guidance provided by USAID/Jamaica to implementing partners. This section focuses on activities falling within the three key Intermediate Results (IRs) established for the ARH program – activities designed to improve access to ARH services, activities designed to improve knowledge and skills related to ARH and activities designed to bring about the implementation of national policies and guidelines in support of ARH. The next section, Project Management, addresses the quality and effectiveness of The Futures Group’s (TFGI) management of the Youth.now project. This is followed by a section on Program Sustainability, which examines sustainability of the program in the context of key institutional constraints and potential of the Ministry of Health (MOH) and other Government of Jamaica (GOJ) ARH implementing agencies. Findings and Recommendations, for both mid-course corrections as well as long-term directions, are a critical part of this evaluation and are included as appropriate within each of the foregoing sections. The final Conclusions section summarizes the degree to which the program is meeting its stated objectives based upon the forgoing analysis. Special note needs to be made of HIV/AIDS prevention and control and how this important activity is addressed in this evaluation. USAID supports HIV/AIDS prevention and control activities in Jamaica both through Youth.now as part of its integrated ARH approach to adolescents, as well as through a much larger and more comprehensive categorical HIV/AIDS project implemented through the MOH and other CAs (HIV/AIDS Prevention and Control Project). The current SO #3 Results Framework includes indicators related to HIV/AIDS at the SO and IR levels. However, Youth.now’s impact on HIV/AIDS at the SO level is negligible, and the project reports results only at lower indicator levels. Youth.now HIV/AIDS activities consist primarily of information programs directed at relatively small numbers of youth (with the exception of its periodic mass media campaigns, which do reach a national audience with HIV/AIDs prevention messages), in contrast with the aforementioned USAID HIV/AIDS Prevention and Control Project, which targets adults and high-risk groups such as prostitutes and men who have sex with other men. The fact that both the ARH/Youth.now and HIV/AIDS Prevention and Control Projects fall under the same SO and are integrated within the same RF has created some confusion regarding accountability for reporting on results. USAID plans to revise the current results framework to create two entirely separate IRs under the SO for ARH 9 and HIV/AIDS, each with its own set of lower level results indicators. This will help clarify the degree of accountability for HIV/AIDS results of the ARH program in the future. II. BACKGROUND A. The International Experience with ARH Adolescence is a period of dynamic physiological, psychological and social change, representing the transition from childhood to adulthood. It is experienced differently in every society. Adolescence is also typically the period when young people begin to engage in sexual activity. Adolescence is generally a healthy time of life, but young people are often at a disadvantage with respect to information about sexuality and access to contraceptive services. In most developing countries, few youth use reliable contraception. Moreover, young people are not always able to make decisions for themselves about their sexuality, since many youth experience coerced sex. Over the last decade, an international consensus has emerged that young people need expanded information, skills and services relating to healthy sexual development and behavior. This consensus has been driven to a large extent by perceived increases in premarital sexual activity and/or pregnancy, and by high rates of HIV infection in some countries among adolescents, especially girls. However, many developing countries—and some developed countries— continue to have a high degree of societal discomfort with premarital sexual activity, and adolescent sexuality is typically a sensitive and politically charged subject. Adult gatekeepers, both in their private roles as parents and their public roles as policymakers, teachers, community leaders and health workers, are often the greatest barrier to creative and effective programs that reach large numbers of young people with education and services relating to sexuality and RH. Beyond access to RH information and services, a multiplicity of contextual social factors – including parents, peers, schools, and other institutions and community factors can have an impact—positive or negative—on young people’s decisions and actions. While these contextual influences appear very important in enhancing protective or risk behaviors, they vary greatly across societies and are not always easily amenable to program intervention. Few large-scale programs have attempted to address these influences in a deliberate way to date. While the evidence from rigorous program evaluations remains very limited, key findings from such research suggest the following: · In general, programs appear more effective in influencing knowledge and attitudes than behaviors. · School or other curriculum-based programs that apply accepted best practices are effective in influencing knowledge and attitudes, and can delay sexual activity and increase use of condoms and contraception, at least in the short-term. · Making clinics youth-friendly has generally failed to increase utilization by young people, except in a few instances where programs have included strong linkages to the community. Multi-purpose youth centers also do not appear to increase young people’s use of RH services. 10 · Mass media is an effective way to reach adolescents and influence their knowledge, attitudes and norms, but links to more personalized activities appear to be needed for behavior change. Although the evidence is thin, linked mass media and social marketing has high potential reach and appears promising. Beyond the limited scientific evidence, a consensus based on program experience suggests the following best practices in addressing young people’s sexual health. · There is a need to program differently for young people according to their different characteristics and life context, especially their age, gender and sexual experience. It appears important to reach young people early, before first sex. · Building skills, both generic life skills and skills specific to sexual risk reduction—such as negotiation of condom use—should be a core intervention for adoption of safer behaviors. · Given the reluctance of young people for a variety of reasons to use clinical services, programs need to move beyond clinics. They should use a greater variety of settings and providers, both private and public, clinical and non-clinical, that build on where young people are already going for reproductive health information and services. · Addressing community norms relating to early sexual activity and multiple partners is important, especially to reduce HIV prevalence among young people. · Given the scale of the challenge, programs need to pay attention to resource constraints and to make the most of existing networks and infrastructures in order to reach many more young people in new and flexible ways. · Perhaps most importantly, linking different interventions in a comprehensive approach appears to increase effectiveness, given the diversity of youth and the multiplicity of influences on young people’s behavior. This assumption is currently being empirically tested in several sites. B. ARH in the Jamaican Context Recent international conferences have served to place the issue of adolescent reproductive and sexual health on the national agenda of countries around the world. In the Caribbean, including Jamaica, efforts have been intensified to implement programs and policies to enhance the reproductive and sexual health of adolescents, whose vulnerability to early sexual activity, unplanned pregnancy, sexually transmitted infections along with other issues such as substance abuse is of significant concern. In 2001 the Jamaican population was estimated at 2.6 million of which the 10-24 age group represented 28 percent. Adolescents 10-19 years old, who are the main target group of the USAID/Jamaica ARH program, accounted for 19 percent. The participation of adolescents in education and employment are two important factors that determine how they can best be reached with ARH information, education and services. In Jamaica, school enrollment rates are high compared to other developing countries. In 2001, gross enrolment rates at the primary and secondary education levels were 99 and 75 percent respectively. The latter enrolment increased from 64 percent in 2000. Although most youth (98% of 12 –14 year olds and 83% of 15-16 year olds) are in school, significant gender differences in enrollment emerge by age 17, at which age only 40 percent of males compared to 57 percent of females are still in school. 11 With respect to employment, in 2001 there were significantly more males than females in the labor force, and also more youth in the older, 20-24 year age group. Both the 14-19 and 20-24 age cohorts recorded declines in labor force participation of 9 percent and 1 percent respectively. It is thought that these declines may be related to the increase in enrolment of young persons in formal education /training institutions. In Jamaica, initiation of sexual activity occurs very early, increasing the risks of pregnancy, sexually transmitted infections and school-drop-out among teens. The 1997 Reproductive Health Survey (RHS) reported the average age of first intercourse among 15–19 year olds who had ever had sex as 15.9 years for girls and 13.4 for boys. This early timing of sexual initiation is supported by numerous other surveys; in addition, both the RHS and some other recent surveys suggest the age of sexual initiation may be even lower among the younger cohorts—12 years for boys and 14 for girls, The Adolescent Condom Survey (ACS) 2001 and the Behavioral Surveillance Survey (BSS) 1999-2000 also confirm high levels of sexual activity among youth. According to the ACS, 74 percent of 15-19 year olds and 9 percent of 10-14 year olds had engaged in sexual activity, with significantly higher levels among boys compared to girls. By age 16, 82 percent of boys and 56 percent of girls were sexually active. The RHS 1997 also found that during the years 1993-1997 the fertility rate for 15-19 year olds increased significantly compared to women 20–24 years. The proportion of births to adolescents has remained virtually unchanged at approximately 30 percent of all births since the 1970s. Of those pregnancies that occurred to women 15-19 years old, 36 percent were planned, 46 percent were mistimed and 18 percent were unwanted, highlighting the need to improve contraceptive protection for women in this age-group. The HIV epidemic in Jamaica is still largely concentrated among high risk groups such as sex workers and men who have sex with men, although HIV infection appears to be increasing in the general population in Kingston and some tourist centers. National adult prevalence is estimated at 1.2 percent; seroprevalence for the age group 15-24 years is estimated to be slightly lower, at 1.03 percent for females and 0.98 percent for males (UNAIDS 2002. Report on Global HIV/AIDS Epidemic 2002). However, the majority of reported AIDS cases are in the age group 20-39 suggests that many infections occur in the adolescent and young adult years. Attitudes, knowledge and skills related to sex and reproductive health are important determinants of adolescent sexual behavior. Research findings indicate high knowledge regarding contraception among adolescents, but there are variations in specific knowledge by gender and age. The 1998 Jamaica Adolescent Study found that among 7th graders, 49 percent of boys and 66 percent females did not know that pregnancy is possible at first intercourse. A 2001 study found that while knowledge about HIV/AIDS was high among 10-19 year olds (87% or higher), the 10-14 cohort had less knowledge of STIs than 15-19 year olds and that the males had less knowledge than females. With respect to the availability and use of contraceptives by adolescents, the 1997 RHS indicated that 65 percent of sexually active 15-19 year old women were using a contraceptive method. The condom was the most popular method (29.6 percent prevalence) followed by oral contraceptives (22.3 percent) and injectables (6.3 percent). Of young men aged 15-19, 82.6 12 percent reported using a method, with condoms again the most popular method (72.3 percent prevalence.) The 1999-2000 BSS also found very high levels of condom use among youth in Jamaica. Among in-school youth, 72 percent reported using a condom at last sex with a commercial partner, and 53 percent reported consistent condom use with a non-commercial partner over the previous three months. For out-of-school youth, condom use was significantly lower, at 58 percent at last sex with a commercial partner and 34 percent consistent use with a non￾commercial partner in the previous three months, According to the ACS 2001, 97 percent of boys and 77 percent of girls aged 15-19 who used a contraceptive the last time they had sex used the condom. The preferred and most used source of condoms for young males was the small shop, while sexually active females depended on their partners to obtain condoms. In addition, youth aged 10-14 said condoms were difficult to obtain because retailers would not sell condoms to them. Factors restricting adolescent access to reproductive health services include legal and operational policies relating to age of consent for both sexual activity and medical services, judgmental attitudes of health providers and other key community level gatekeepers, and the poor quality of clinical services. The current age of consent in Jamaica is 16 years and adulthood is age 18, resulting in legal constraints in the provision of health services to persons under the age of 16 years. There is policy support by the Ministry of Health based on the so-called Gillich Law which states that persons below the age of consent as established by law should have access to social, medical information and counseling on reproductive health. However, this policy is often ignored in practice. Nonetheless, in 2001 10-19 year olds made a total of 22,250 family planning visits to MOH clinics, accounting for roughly 2500 new clients (this excludes postnatal acceptors.) In addition, the 10-19 age group accounted for one quarter of total visits to antenatal clinics, and 21.9 percent of postnatal visits. Twenty three percent of the latter became new family planning acceptors with the method of choice as follows; 26.7 percent oral contraceptives, 23.6 the injection, 28.9 percent intrauterine device (IUD), 17.1 percent the condom; 53.0 percent opted for the dual method. Over the last three decades there have been several interventions in the public and non￾governmental sectors (NGOs) supported by international funding agencies to improve reproductive health services for adolescents. These interventions have focused on the provision of education and information and services to adolescents through Ministry of Education, Health, Agriculture and Youth and Community Development channels including schools, health centers, and youth clubs. During the decade of the 1990s, such interventions became more specifically focused on adolescents. Several adolescent reproductive health projects initiated in recent years with international assistance from UNICEF, UNFPA and USAID (the largest) aimed at improving reproductive health knowledge and access to services, with the aim of reducing adolescent fertility. USAID began funding family planning and reproductive health programs in Jamaica with adolescents as a key target group as early as the late 1970s. Significant levels of support were 13 provided during the 1980s to the National Family Planning Board (NFPB) for both Information Education and Communication (IEC) and service delivery programs as well as to local NGOs such as The Women’s Center and Operation Friendship. During the last decade, USAID provided support for three major activities under the Strategic Objective (SO): “Young Jamaicans Better Equipped for the 21st century. The three activities funded by USAID are: · AIDS/STD Prevention and Control activities (1988-2001) · The Family Planning Initiative Project (FPIP 1991-1999) · The Uplifting Adolescent Project (UAP 1996– 2000). These interventions were focused on: prevention and control of HIV/STDs including condom social marketing to increase non-traditional outlets, behavior change communication, mass media, and surveillance; strengthening family planning services; and providing a range of services to “at risk” youth (10-14 years) to encourage their continued education. Of these three projects, only FPIP had a specific reproductive health focus that directly targeted adolescent sexual behaviors and practices. C. Summary of the USAID/Jamaica ARH Program The Jamaica ARH Program was designed to address gaps in the reproductive health program implemented by the MOH and the NFPB. Gaps identified included limited dissemination of a definition of reproductive health, lack of a reproductive health policy, limited impact evaluation of RH activities, inadequate family life education (FLE), the need for clinic friendly services and limited computerization of health data. The program is being implemented over the period 2000–2004 at an estimated cost of $12.7 million. Most of these funds are dedicated to a contract with The Futures Group International (TFGI) for the Youth.now project ($8.97 million) while the remainder is programmed separately by USAID/Jamaica through USAID/W worldwide cooperative agreements with agencies such as Family Health International (FHI) to support a range of complementary ARH activities. The Ministry of Health was identified as the key vehicle for project implementation, given its position as the major provider of health-oriented education and services and its collaboration with other institutions needed to facilitate a multi-sectoral approach to address important non￾health factors associated with ARH. Youth.now offices are located in the MOH. Organizationally, the project falls under the Health Promotion and Protection Division. The principal and strategic objective of the program is “Improved Reproductive Health of Youth”, and there are related IRs that are designed to support its achievement. The principal IR (IR 1) is “Increased use of quality reproductive health and HIV/STI services and preventive practices”. Indicators have been established at the SO level which include reductions in youth fertility, HIV seroprevalence among STD clinic attendees and HIV seroprevalence among antenatal clinic attendees. As discussed above, the latter two indicators related to HIV are addressed primarily through a separate HIV/AIDS Prevention and Control Project with the MOH. Indicators at the IR 1 level include the number of new family planning acceptors, the number of users of other RH services and the number of youth practicing low-risk behaviors. Falling under IR 1 are three lower-level IRs that relate to increased access to quality RH and 14 HIV/STI services, improved knowledge and skills related to RH and HIV/AIDS/STIs and the implementation of national policies and guidelines in support of RH with a focus on youth. In order to achieve the above results, nine implementing partners with specific and complementary roles and responsibilities were enlisted. The implementing partners are as follows (note that MSCI, JHPIEGO and DCC are under sub-contract to TFGI, while the remaining CAs are funded directly by USAID/Jamaica): · The Futures Group International (TFGI)– The institutional Contractor for the ARH Project in collaboration with the Ministry of Health · Margaret Sanger Center International (MSCI)– Under sub-contract to TFGI to deliver in￾service training to providers · Dunlop Corbin Communications (DCC) – Under sub-contract to TFGI to develop and implement mass media and public relations programs · JHPIEGO Corporation/Training in Reproductive Health III – Under sub-contract to TFGI to conduct pre-service training for nursing schools and the Nursing Council of Jamaica · Family Health International (FHI)/Contraceptive Technology Research – Undertake a Behavioral Surveillance Survey (BSS); carry out parent education, and train MOE school guidance counselors utilizing ASHE; develop client appointment system and improve counseling for contraceptive users · Academy of Educational Development/CHANGE – Carry out a study of youth resiliency and assets · Academy of Educational Development/LEARN Link Project—Strengthen MOH management information systems · Deloitte Touche Tohmatsu/Commercial Market Strategies –Carry out an adolescent condom survey and improve condom access through the commercial sector · TFGI/POLICY II– Support the MOH Strategic Framework for Reproductive Health (strategic planning at regional and parish levels), conduct a Policy Environment Survey (PES) and support a multi-sectoral approach to adolescent policy development through the National Center for Youth Development (NCYD) · University Research Co-operation/Quality Assurance Project – To develop a sustainable communication strategy and standards, certification and accreditation for Youth Friendly Services (YFS.) In addition to the above, in 2001, in response to a request from NFPB, USAID/J provided a grant to the Centers for Disease Control (CDC) to assist the Board in carrying out the 2002 version of the National Family Health Survey. Among the above implementing partners, TFGI’s Youth.now project undertaken in collaboration with the Ministry of Health is by far the largest ARH activity. Key Youth.now interventions include: · Establishment of models for delivery of ARH information and services · Training and capacity building · Targeted behavior interventions · Public education 15 · Policy development and advocacy; and · NGO support and small grants. Under the NGO support and small grants program current recipients include FAMPLAN, Women’s Center Foundation of Jamaica, Jamaica Foundation for Children, YMCA – Kingston, Children First and Jamaica AIDS Support. Recent awardees include Children in Community for Change and the Louise Pitter Clinic, North Street United Church and pending are grants for Girl Guides Association and Whole Life Ministries (See Annex D for details on NGO subgrants). Other non-Youth.now CA activities were designed to inform the development of the above interventions, as well as to expand and deepen support for the overall ARH effort. III. PROGRAM PERFORMANCE The following section details the evaluation team’s assessment of key program interventions to improve access to reproductive health services for adolescents, knowledge and attitudes among adolescents related to reproductive health and the overall policy environment needed to support reproductive health. As will be noted and discussed below, the findings of the team with respect to some key interventions have been positive, while the results of others have been disappointing. Program training activities, for example, have been enthusiastically received at all levels, and there appears to be some evidence that this training is beginning to have a positive multiplier effect. On the other hand, utilization by teens of MOH YFS sites for contraceptive services has been almost negligible to date, which calls into question whether that approach should be abandoned in favor of a more broad-based effort to reach teens in more diversified and non-clinical settings. A. Fertility and Contraceptive Use 1) Findings: a) The direct impact of ARH program activities to date on lowering fertility and increasing contraceptive use in Jamaica is negligible. It is possible that the long￾term impact of Youth.now efforts will have a better result, but the current evidence suggests this is unlikely. b) Youth.now interventions in target communities and parishes do not appear to be having the desired effect on increasing contraceptive use among teens. c) There are other interventions that could possibly have more direct impact on increasing adolescent contraceptive use that are not currently part of the ARH program. 16 2. Discussion: The Strategic Objective level result the ARH program hopes to reach is improved reproductive health of youth as measured by age-specific fertility rates of adolescents. Two other indicators dealing with HIV/AIDS seroprevalence among STD and antenatal clinic attendees are included at this level, but as was discussed in the background section of this report, they are not addressed herein since they are the province of USAID’s separate HIV/AIDS Prevention and Control Project. Suffice it to say that the latest seroprevalence rates among all antenatal clinic attendees stood at 1.15% and this rate appears to be decreasing slightly. Adolescents comprise about a quarter of all antenatal patients in MOH hospitals and clinics. In addition, seroprevalence among STD clinic attenders stood at 6.9% in 2001 with adolescents comprising a relatively small percentage of the total. However, as was noted previously, adolescents comprise a rapidly growing segment of the population infected with HIV, and are vulnerable to infection. HIV/AIDS messages are an integral component of ARH training, education and counseling programs, though given the relatively small size of the ARH program, the impact on seroprevalence at the national level is likely minimal. With respect to impact of the ARH program on age-specific fertility rates, the direct result is also likely to be minimal due to the relatively small number of teens reached through the program, and the even smaller number who alter their fertility practices as a result. The last national Family Health Survey (FHS) that measured adolescent fertility was carried out in 1997. At that time the age-specific fertility rate among 15-19 year olds was 112 per thousand, which does not represent a statistically significant increase over the rate of 107 per thousand, recorded in the previous 1993 national survey. Fieldwork for the current 2002 round of the FHS is now underway, but the preliminary results will not be available for at least six months. As in the past, the ARH program provides partial support for the FHS, both directly to the National Family Planning Board (NFPB) and the U.S. Centers for Disease Control (CDC). Given the importance of this highly regarded survey to Jamaican and donors alike for policy and program planning purposes, these are funds very well spent. The indicators for IR 1: Increased use of quality reproductive health and HIV/STI services and preventive practices, described in the USAID Results Framework (RF) and the Youth.now Reports Monitoring and Reporting Plan (RMRP) are numbers of new acceptors of family planning/contraceptive services and number of users (RF) or visits (RMRP) of other reproductive health services at YFS and other sites in target communities and parishes. The RMRP establishes a target of 250 monthly visits for RH visits, including HIV/STI visits by the end of the second planning period, September 30, 2002. Unfortunately, due to problems hindering current and accurate collection and reporting of service statistics at YFS sites, it is not possible to determine whether this goal is being met. Moreover, the lack of clear definition of what constitutes RH for reporting purposes also makes it difficult to arrive at a credible number. If one accepts a broad definition of RH and includes teens returning repeatedly to sites for group “counseling” sessions, the target is probably being met. However, if one accepts a narrower definition and includes only contraceptive users and not visits, it is probably not being met. Regarding new acceptors of contraception at YFS sites, The RMRP has not yet established specific targets. However, the judgment of the evaluation 17 team based upon a review of the available data and conversations with clinic staff at YFS sites is that the number of new acceptors of family planning services is negligible. For example, the FAMPLAN clinic in downtown Kingston reports only “a few” family planning visits per month and the Balaclava site around 8-10 a month. These data are for family planning visits, not new acceptors. Nor do we know from the currently available information what the previous intake of new family planning acceptors was, in order to assess growth due to the intervention of the project. Despite the disappointing performance of the project thus far, as measured by contraceptive use, it is possible that the long-term impact of Youth.now’s efforts could be much greater. For example, it may be (despite the impression given the evaluation team to the contrary) that Youth.now peer counselors in public schools are generating more contraceptive users among their classmates than are currently being reported, since adolescents are often reluctant to disclose their sexual behaviors to their peers. Also, it is still relatively early in the history of the YFS clinic sites and it is possible (although not likely) that contraceptive users at the sites could increase dramatically. It is also possible that the new found enthusiasm for ARH programs among the MOH and other providers could have a multiplier effect that will extend well beyond the project. And, it is possible that the strong abstinence message of the project could result in a large number of teens that delay sexual initiation and avoid early pregnancy. However, given the limitations of the current program along with the weak enabling environment in Jamaica, it appears unlikely that the program will have a significant impact on contraceptive use and adolescent fertility. If USAID wishes to support efforts to have a greater impact on lowering high adolescent fertility, the 1997 RHS along with the more current BSS and CMS surveys provide some insights as to potential alternatives. Given the heavy reliance of Jamaican youth on commercially marketed condoms, new approaches to reaching sexually active teens are called for, especially through the private and commercial sectors. As noted earlier, condom use among sexually active teens is already very high. However, about 18% of in school and about a third of out of school youth had sex unprotected by a condom in the three months prior to the survey. Thus, while there is room to more vigorously promote condom use among adolescents and especially out of school youth, finding ways to generate more consistent condom use is even more important. In addition, Jamaican youth tend to have births early. According to the FHS, about 11 percent of 15-17 year olds and 34 percent of 18-19 year olds had a first birth. Yet very few of these teens are using more reliable methods of contraception. Much more could be done to support the use of IUDs and hormonal contraception among this group of childbearing teens and among older teens in general. Finally, since recent data indicates that teens are becoming sexually active at an even younger age (age 12 for boys and 14 for girls) attempts by Youth.now to reach younger teens with abstinence/delay messages seem well placed. However, in the Jamaican context, the abstinence/delay message needs to be complemented by a stronger emphasis on expanding access to contraceptive services for already sexually active teens. Moreover, as currently delivered there is a concern that the abstinence message may constitute a barrier to contraceptive use by these sexually active teens. 18 3. Recommendations: a) In order to increase impact on contraceptive use, USAID should fund efforts to more directly reach sexually active adolescents with condom promotion programs using private and commercial channels, as well as programs to reach older teens with more effective methods of contraception including IUDs, hormonal methods and emergency contraception. b) Youth.now and the MOH should redirect the project’s efforts away from special clinical sites for adolescents in favor of broader, youth-friendly training programs for providers and community opinion leaders. B. Increased Access to Quality Reproductive Health and HIV/STI Services “It’s easier for teens in Jamaica to get an abortion than to get contraception” —Key Informant interviewed by Evaluation Team This section of the report addresses the progress and performance of the ARH program with respect IR 3.1.1, Increased Access to Quality Reproductive Health and HIV/STI Services. Three main activities are addressed: Youth.now’s effort to establish “Youth-friendly Service Models;” project subgrants to non-governmental organizations (NGOs); and the collaborative effort between Youth.now, QAP and the MOH to develop standards and certification for youth-friendly services and a certification process. MOH Youth Friendly Service Models 1. Findings a) Youth.now’s team approach to training staff at the youth-friendly sites has deepened the receptivity and commitment of health staff to serving adolescents, and helped to strengthen school-health center linkages. b) The investment in youth-friendly service models has resulted in a few scattered sites serving very small numbers of young people with contraceptive services. The volume of services is unlikely to increase significantly over the life of the project, given the reluctance of youth to use public sector RH services. c) Youth.now’s evaluation plan is not adequate to test the relative impact and effectiveness of the various models for delivering youth-friendly services. There is also a lack of clarity as to how broader educational outreach should be treated in tracking ARH “services.” 19 d) The development of model sites, especially within the public sector, is not a productive strategy for expanding access to ARH services. It does not merit the current level of effort and should not remain a primary focus. 2. Discussion The FGI contract calls for Futures to “test” different “models” for providing RH services to adolescents. Youth.now has worked extremely hard to establish “Youth-Friendly Services” at seven sites in five parishes, four with the MOH and three with NGOs. · In St. Elizabeth, the project is implementing school-linked programs with MOH health centers at Balaclava and Junction. · The May Pen health center has sponsored a weekly educational and recreational program, INFO-FUN, for youth from local schools. · In Kingston, Youth.now recently launched an adolescent clinic at Glen Vincent, a freestanding clinic in a central, commercial area. · NGO model sites include Children First, an NGO based in Spanish Town, St. Catherine, which links RH activities to youth employment activities, and the YMCA/Kingston, which is to provide counseling, condoms and referrals, as well as RH education. A grant to FAMPLAN also includes support for clinic-based services. It is noteworthy that there are some differences in the interpretation of Youth.now’s mandate with respect to these “models.” Youth.now is very focused on the specific model sites, while USAID and the MOH articulate a need for broader support to improve services for adolescents and more rapid rollout at the parish level. Implementation: At the MOH sites, health staff have been trained to provide “youth-friendly services;” peer counselors or other educators and outreach personnel have been recruited and trained; support has been provided for some limited facility enhancements; and Youth.now provides technical support and monitoring on an ongoing basis. Youth.now has also convened a Youth Friendly Services Working Group to define a core package of ARH services that includes sexuality education, counseling, condoms and other contraceptives, pregnancy testing, STI counseling and syndromic management, and referrals for other special services including pregnancy-related care. In actual implementation, not all seven sites are fully operational or offer the full package of ARH services. FAMPLAN has experienced long delays in renovating its clinic to accommodate adolescents, and the Glen Vincent site only recently opened. The YMCA and Children’s First, both recent grantees, are not yet providing all anticipated services, and as such, these NGO sites have limitations as an alternative model for delivering ARH services. At most public sector sites, “special” adolescent sessions are held only once or twice a month, often at times that overlap with school hours and are not convenient to young people. RH services can also be accessed during routine outpatient hours, although nurses see dedicated adolescent sessions as offering more privacy and allowing them to spend more time counseling young people. As such, the “models” described by Youth.now staff are not clearly distinguishable from each other on the ground. 20 Impact on MOH Staff: At the better-established public sector sites (Balaclava, Junction, May Pen), the team approach to training staff appears to have generated a critical mass of enthusiasm and commitment to serving adolescents. While the team heard many comments that current commitment to ARH at each site is too dependent on individual personalities, interest at the sites visited appears relatively broad-based. Providers report that training has made them more open to serving adolescents. Some MOH staff who participated in the Youth.now training have taken the initiative to establish special activities for adolescents. In May Pen, following a loss of momentum in the Youth.now￾sponsored INFO-FUN recreation program, staff recently initiated a special adolescent session at the health center, without Youth.now assistance. At Junction, health center staff have initiated “hot-line” telephone counseling for adolescents. Given that provider attitudes are notoriously hard to change and MOH staff are seriously overburdened, these are indicators of significantly improved receptivity and deepened commitment to serving adolescents. A noteworthy project innovation is the inclusion of dental nurses in the RH training, taking advantage of a non-stigmatized health service used by large numbers of adolescents to provide RH information, counseling and referral. (Annual check-ups are required for Fall school enrollment.) Many of the dental nurses appear very proactive, displaying penis forms and using them as an entry point to discuss and demonstrate condom use. At the project sites, Youth.now has also helped strengthened links between health centers and schools. Nurses are reaching out to the schools and community, including parent and community groups. At Junction, they spend one day a week in the schools, where the school guidance counselor “gives” the nurses her session for RH discussions. At Balaclava, once a month, groups of students are brought in by their guidance counselors, teachers and school nurses for RH “talks” and videos, and for tours of the center. Youth Involvement: Youth.now deserves credit for its efforts to involve youth at model sites through linked peer programs. However, the experience with peer education is mixed. The experience has been very positive for a select group of youth trained as peer educators, particularly those who participated in a highly regarded one-week residential training, and those youth at the May Pen INFO-FUN program whose involvement has been especially intense. This latter group was by far the most dynamic, articulate and engaged of the peer educators interviewed by the team. However, it is questionable how broadly or successfully they have reached out to other, more vulnerable young people, especially since many peer educators themselves appear judgmental and advocate abstinence for all teens, whether already sexually active or not. The May Pen experience reflects some of the classic pitfalls of peer programs. A group of 25-30 youth leaders who participated regularly in the weekly INFO-FUN educational program appear to have developed an elite profile that did not lend itself to broader participation and attract other youth to the program. The project is also now facing the turnover that is routinely experienced in youth peer programs, as many of the original core group have graduated while others have moved on to new interests. Because of the failure to plan for recruitment of new cohorts on an 21 ongoing basis, INFO-FUN has not convened since the start of the school year in September. Health staff at May Pen note they have learned valuable lessons about the need to expand coverage and recruit younger youth in order to move forward on a more sustainable basis. Sustainability: With respect to costs, training of staff has been the key intervention at the sites. (Sustainability issues relating to Youth.now training activities are discussed elsewhere in this report.) For the most part, dedicated adolescent clinic sessions and school-health center linkages do not appear to require significant additional inputs, with the exception of staff overtime for services provided outside normal clinic hours. However, in terms of sustainability, another key input has been the support that Youth.now core staff and parish officers provide to both health staff and peer educators. It is clear that at MOH YFS sites in both St. Elizabeth and St. Catherine, the involvement of local youth would not exist without the aggressive outreach and recruitment carried out by Youth.now staff. Impact on Services: Despite some positive results, very few adolescents are accessing RH contraceptive services at the model sites. A substantially larger (but undetermined) number are served through project outreach and educational efforts. At Junction, about 10 youth a month are coming for FP services; at Balaclava, staff reported 78 visits (including repeats) by 10-19 year olds from March to October 2002. FAMPLAN had about 82 adolescent visits in the second quarter of 2002. No data are available to indicate if these numbers represent an increase in adolescent visits following the Youth.now interventions. Only a handful of youth sought individual counseling at these sites, and visits for STI care are not disaggregated from overall visits for curative services. While larger numbers are reported at the public sector sites for general health and dental services, these latter visits are presumably unrelated to Youth.now efforts. Thus, it appears that the youth-friendly sites have not been able to overcome young people’s reluctance to utilize public sector RH services. Peer educators note that youth do not like to be seen in public waiting areas, and prefer to travel to health centers in another localities for RH services. While health staff who have been trained report significant improvements in their own attitudes, they almost uniformly report that in RH counseling sessions with teens they begin by advocating abstinence, apparently even for sexually-active older youth. Judgmental provider attitudes may thus also continue to deter young people from coming to public sector clinics. In sum, too great an effort has been invested in developing a few scattered sites which provide RH services to a small number of adolescents. Despite the relatively short period of implementation, the effort as currently conceptualized to develop youth-friendly models, especially within the public sector, does not appear to be a productive strategy for reaching more young people with RH services. It is possible that more services are being provided than evident from MOH reports. Official clinic statistics do not fully capture condoms “unofficially” distributed by peer educators or dental nurses, or contraceptives purchased from private sources as a result of peer counseling or other educational efforts. To date, Youth.now has not systematically tracked RH service statistics at youth-friendly sites, including the “baseline” prior to project intervention. However, it is now introducing improved reporting formats for both clinic and outreach activities. 22 Evaluation: Youth.now has spent a great deal of effort and generated substantial data in support of its evaluation efforts. Given the intent of the project to test new models and approaches to reaching adolescents with RH services, such attention to evaluation of project interventions is highly appropriate. However, although operations research is underway at one model site, control sites are not included in most instances, and baselines have not always been conducted. Youth.now could also have incorporated more use of creative approaches such as “mystery clients” to evaluate provider interactions with adolescent clients and adolescents’ perceptions of quality of care. A major weakness is that Youth.now has not had an overall plan for assessing the relative impact and effectiveness of the various YFS model sites, or for analyzing their replicability and sustainability in the event they were to prove successful. What constitutes “success” is also unclear, since the Futures contract emphasizes increased contraceptive use, while Youth.now includes educational contacts in ARH “services.” Future Directions: The priority is to diversify services for adolescents. Internationally, the evidence is still thin on what works in developing countries to increase the use of RH services by adolescents. But the potential in Jamaica to expand access through a broader array of private sector and non-clinical channels is supported by a host of research studies. Condoms are the most widely used method among adolescents in Jamaica, and the ACS 2001 reports that nine of every ten boys who buy condoms obtain supplies from private retailers, especially small groceries. While USAID is planning to work through CMS to expand access to condoms and emergency contraception through the private sector, the scope of these activities are currently very limited. USAID should more substantially engage the private sector in light of its current role in serving youth. Such collaboration needs further exploration, but could potentially involve training for pharmacists and shopkeepers to improve access to condoms for younger youth and girls, or “social franchising” (branding) of youth-friendly outlets. Another priority should be to work with networks of pharmacists and private doctors to improve access to STI diagnosis and treatment for adolescents. At the same time, there remains an important role for the MOH in ARH. In Jamaica, public sector clinics are important for youth to access hormonal and other clinical contraceptive methods, which have the potential to play a larger role in teen pregnancy prevention. USAID should expand support to the MOH for hormonal and other clinical contraceptive methods. Adolescents also need other clinical services such as STI and pregnancy-related care. A small number of adolescents already use MOH contraceptive services; in 2001, 10-19 year olds made a total of 22,250 family planning visits to MOH clinics and accounted for roughly 2500 new clients, excluding postnatal acceptors who are many more in number. Over the long-term, the entire MOH cadre should undergo basic training and sensitization in the provision of youth friendly services, even though this may not translate in the short-term into a significant uptake in clinic services by adolescents. 23 Youth.now, through its training activities, could potentially support the MOH in improving ARH services in a more broad-based way. Discussions currently underway with the Southern Region to expand and institutionalise youth-friendly services through training of supervisors and community-based health care providers already represent a move in this direction. Such training could be linked to limited follow-up technical assistance to help health teams make simple improvements in clinic services to address barriers to access by youth. As discussed elsewhere, Youth.now should continue the team training approach while revising the curriculum to underscore the importance of more readily providing contraceptives, especially for older and sexually active youth. Such an effort will require a counterpart commitment by the MOH to integrate YFS training into its ongoing training efforts, and to more vigorously promote the provision of ARH services. While individual clinics will need to develop outreach and promotional strategies at the local level, a broader effort could eventually lend itself to promotional support through media and other national channels, as discussed later in this report. The regional health authorities should also encourage health center staff to further strengthen their links with schools. While existing peer counseling networks in schools are a logical base for RH education and outreach, more attention is needed as to how high quality training in RH for peer counselors can be institutionalized, and how such efforts can achieve broader reach and sustainability. As with health staff, future training for peer counselors needs to better address their own personal biases. 3. Recommendations a) The USAID ARH program needs to more vigorously engage with the commercial sector and private health providers, especially those in non-clinical settings. More funds should be allocated for this purpose. b) USAID should explore expansion of support to the MOH to promote the use of hormonal and other clinical methods of family planning for adolescents in MOH clinics, through a more strategic, segmented approach that recognizes the needs of different subgroups of youth, including post-partum teens. c) For the last two years of the project, Youth.now should shift the focus away from the model YFS sites in MOH clinics to a scaled-up training effort to sensitize local health teams to the importance of serving adolescents and strengthening links with schools. d) Youth.now will need to revise its M&E plan to incorporate meaningful but practical measures of the impact of these changes in direction. USAID and Youth.now should also agree on what constitutes “ARH services” for purposes of performance monitoring. 24 NGO Support and Small Grants 1. Findings a) The small grants to NGOs reflect a laudable effort by Youth.now to reach out to more vulnerable, out-of-school and inner city youth, but are serving only small pockets of these hard-to-reach populations. b) The limited coverage and weak capacity of these NGOs undermines their effectiveness as a vehicle for delivering RH education, and their potential as a viable alternative channel for providing RH services. 2. Discussion As the time of the evaluation, Youth.now had awarded eight small grants to youth-serving NGOs, some of them mentioned above. Grants to the Women’s Centre Foundation of Jamaica (WCFJ), FAMPLAN and the Jamaica Foundation for Children (JFC) have been implemented for about a year, while grants to Children’s First and the YMCA have been in effect for about six months. Three other grants had just been awarded and two new grants were pending at the time of the evaluation. These grants, which often cover a multi-year time-frame, vary in size from roughly JA $1.1 million (US $23,000) to JA $ 9.6 million (U.S. $204,000), with the average around JA $4.6 million (U.S. $98,000.) Annex D summarizes activities supported by the grants active at the time of the evaluation, and their progress to date. The NGO grants primarily support educational interventions, for the most part targeted at specific groups of out-of-school, at-risk youth, with a special emphasis on boys. Activities supported under the grants include: · Training of peer counsellors and support for peer education programs · Community awareness-raising events and outreach · Strengthening of RH counselling within a telephone hot-line service · Parent education. Youth.now is to be commended for reaching out through youth-serving organizations to reach more vulnerable and out-of-school populations. More recent grants also give greater emphasis to HIV prevention. One such grant funds efforts by Jamaica AIDS Support to reach marginalized inner-city youth through music and sports with HIV prevention messages. For the most part, the coverage of these programs remains very limited. Most grants reach fewer than 200 youth on a consistent basis. In many cases, these are static populations, such as those enrolled in the Uplifting Adolescents remedial education program at the NGOs’ sites, and have limited potential for broader coverage. The YMCA and Children’s First are also not yet reaching the broader constituencies anticipated in their project proposals. In addition to group RH sessions, project staff appear to provide extensive one-on-one counseling, raising concerns about 25 cost-effectiveness of project interventions. The NGOs should be encouraged to expand outreach to neighboring communities, or to deliver curriculum-based interventions on a more intensive and “rolling basis” to reach additional groups of youth. The exceptions in terms of coverage are FAMPLAN, which is reaching 1000 youth in selected schools through regularly scheduled RH educational sessions, and the JFC hot-line service. Hot￾line staff have fielded over 2300 calls during the project period, of which a significant number are for reproductive health related problems. The program, staffed entirely by well-trained, committed volunteer youth counsellors, provides a unique and valuable service. If NGO grants exist primarily to serve special “niche” groups of adolescents, one must question whether this is a valid approach given the limited opportunities for scale up of these interventions, More recently, Youth.now has begun discussions with the Social Development Commission (SDC) and police youth clubs regarding potential partnerships with a sports and leadership orientation. These are organizations with island-wide networks that can work at scale, and merit further exploration and support regarding possible collaboration. NGO staff do not appear to be applying accepted best practices in their educational interventions. Research suggests that coherent curricula, interactive methodologies and appropriate instructor training are key to the behavioral impact of RH and life skills programs. In their RH and life skills sessions, NGO staff appear to be drawing in an ad-hoc way on different materials provided by Youth.now or others. Several are developing their own curricula and materials, raising questions about quality, effectiveness and potential duplication of effort. Despite a cumbersome review and approval process, many grants still lack clearly defined goals and activities. Youth.now routinely provides feedback to prospective applicants on their proposals. While the grant agreement should be an important tool to guide implementation, grant documents are often vague regarding geographic coverage and numbers of intended beneficiaries. In a number of cases, proposals include evaluation plans that appear excessively complex for unsophisticated grantees and of peripheral value in evaluating the NGO interventions. Finally, while the primary intent of the grants is to improve young people’s knowledge and skills, linkages to contraceptive services are generally weak. Few NGOs make condoms easily available on site or through peer educators. The few referrals to date have been for general medical services. Many of the NGO grants are relatively recent, and the NGOs have not yet fully mobilized. Many of them are also new to the RH area. Still, given the challenges young people face in obtaining contraception, there are missed opportunities to bridge the gap between improved knowledge and access to services. 3. Recommendations a) Youth.now should not support new NGO grants beyond those that are awarded or pending. It should work with current grantees to modify existing workplans for broader reach and impact, to incorporate more specific objectives, and to simplify evaluation plans. 26 b) Wherever feasible, grants should be modified to include more explicit links to services. Out￾of-school programs should make condoms available on site, while peer programs should have explicit targets for peer contacts and include community-based distribution of condoms, including efforts to increase use effectiveness. c) Youth.now should provide increased technical assistance to grantees in identifying appropriate curricula and materials for educational interventions, and should also monitor the quality of educational programs. d) Any potential follow-on activity should explore more systematic approaches for reaching out-of-school-youth, for example through partnerships with the SDC, police youth clubs, and other networks that have potential for broader reach and sustainability. Standards and Criteria and Certification for Youth-Friendly Services 1. Finding Prospects for institutionalization and enforcement of the standards and guidelines and certification process for youth-friendly services developed by Youth.now with QAP are very mixed. A clinic-driven process that relies on voluntary self-assessment and requests for external certification is unlikely to have teeth and elicit the cooperation of health center staff. 2. Discussion As part of its mandate to develop youth-friendly services, Youth.now has collaborated with the Quality Assurance Project (QAP) in an effort to develop objective standards for youth-friendly services for which the MOH can hold health facilities accountable. Youth.now and QAP have worked together in a complementary manner, with Youth.now taking the lead on the field application and pilot testing of the standards, while QAP has worked with the MOH to lay the groundwork for institutionalizing the activity. Youth.now has worked hard to simplify and streamline the complex “focused accreditation” approach developed elsewhere by QAP and to adapt it to the practical realities of MOH facilities in Jamaica. Youth.now helped orchestrate the pilot testing of the standards at seven sites, and at a follow-up workshop, in the interests of practicality, involved MOH field staff from those sites in narrowing the original ten standards to five. Youth.now also deserves credit for its efforts to engage appropriate MOH staff from the Standards and Regulations department early on, to maximize the chances of eventually institutionalizing such certification within the Ministry. The standards and criteria have been submitted for formal Ministry approval, and the activity is on track with respect to the current workplan. However, prospects for institutionalization and consistent enforcement of the standards over the long-term remain unclear. The Ministry has opted for a process of voluntary certification rather than more formal accreditation, recognizing both the practical difficulties it faces in enforcing mandatory standards and the need for standards for quality of care to extend beyond the narrow 27 area of adolescent health services. The current plan is to move forward with a more clinic-driven, voluntary process of self-assessment and requests for external certification. The downside is that this process will not have teeth unless vigorously supported by regional health management. Health center staff may not cooperate with a voluntary certification process, viewing it as one more burden for which over-worked health staff are held accountable. A significant level of effort is still required for Youth.now to bring this activity to the point of hand-off to the MOH. It is likely that, following MOH approval, the standards and criteria as well as the self-assessment tool will need further revision. A process for certification still needs to be developed with the MOH, and a pool of external assessors will need to be trained. The dilemma is that a significant investment has already been made in this activity, and it is unlikely to move forward without further involvement from Youth.now. At the same time, it is unclear whether certification will serve any meaningful purpose. Moreover, given limited staff resources, continued investment of effort in clinical aspects of adolescent health only serve to further divert the project from potentially more productive investments of effort. On the positive side, the standards and criteria will provide concrete, objective guidance on what constitutes a “youth-friendly” service. The MOH staff person in the Department of Standards and Regulation is enthusiastic about the process, and believes it could be institutionalized within two years. 3. Recommendation USAID, the MOH and Youth.now need to review the relative value of the certification intervention and come to an agreement on an appropriate future level of effort for this activity, keeping in mind both the significant investments to date and the uncertainty as to whether certification will be a viable process and serve a real purpose. C. Improved Knowledge and Skills Related to Reproductive Health and HIV/AIDS/STI The Youth.Now RMRP presents a three-prong strategy for achieving this result. The strategy consists of: (a) a group of activities directed at improving knowledge and skills of providers of care to adolescents, primarily training activities and collaborating with partners to promote Adolescent Reproductive Health (ARH); (b) a group of activities linked to improved knowledge and skills of adolescents themselves in ARH including HIV/AIDS/STI, development of a Behavior Change Intervention (BCI) strategy, conducting mass media interventions at the national level, developing information-education-communication (IEC) materials to support behavior change communication (BCC) interventions at the Parish/community level, and designing and implementing targeted community interventions at the Parish level; (c) a group of activities for liaising with Ministry of Health education/communication/PR staff, and for coordinating a public relations program. For purposes of convenience we will describe the above three prongs as Training and Partnership, BCI/BCC, and Public Relations, and offer an assessment accordingly. In addition to Youth.now efforts, other CAs are providing coordinated support including: (a) the JHPEIGO clinical training and nurses’ training curriculum; (b) FHI and ASHE’s training of parents, teachers and guidance counselors; (c) AED’s Change program 28 dealing with a resiliency and assets approach to ARH; and (d) QAP’s work on a sustainable communication strategy for family planning, which is primarily a rolled-out training effort. 1. Training and Partnership a. Findings · The Youth Now Project has made significant contributions in improving adolescent reproductive health knowledge and skills among service providers and community leaders through training activities in project parishes and regions and through national networks. Although not all providers in the project sites have been trained, it is expected that they will be over the remaining project period. · There is evidence of quality and easily replicable training activities and materials. Curricula and related materials are used regularly to conduct spin-off education and training events by trained master trainers and facilitators among providers, religious groups, parents organizations and men’s networks. · There is evidence of increased ARH awareness and activity resulting from the project strategy to develop skills among gatekeepers in target communities. Not only has demand increased for ARH activities in project parishes but also throughout national networks of churches, parent organizations and men’s organizations, in addition to new demands for training of service providers in Southern and Western regions. · Providers have gained ARH knowledge and skills through training that has been beneficial in improving access to services for adolescents. However, there is still a tendency on the part of trained providers to counsel abstinence as the preferred family planning method. Such messages discourage contraceptive use among older, sexually active teens. · In an effort to strengthen ARH knowledge and skills among nurses over the long term, Youth.Now and JHPIEGO have assisted 20 nursing tutors from five nursing schools to develop an ARH curriculum that will be integrated into the revised nursing curriculum in 2003. Although a revised second draft of the curriculum adequately addresses general reproductive health knowledge and skills for dealing with adolescents, it is still unclear if there is sufficient training time devoted to sessions that impact attitudes and values of nursing students. · The project regularly conducts evaluations of all it’s training programs and plans to carry out further assessments of quality of services delivered by trained providers through analysis of self-reported events conducted by master trainers and facilitators, as well as intercept surveys at service sites. The project intends to measure training effects on gatekeeper attitudes and subsequent actions through the PES survey. Project documents do not indicate when these subsequent evaluations will occur. · Training activities have been designed to create and utilize facilitators and master trainers to support ARH work in the model project’s service sites and associated communities. As the 29 project sites and audiences have expanded and the number of multiplier training events have increased through local and national networks, it is difficult to see how these events are being coordinated and fed back into the project’s overall training strategy. · Currently there is limited training infrastructure at the Ministry of Health and in the Regional Health Authorities where the project is operating. Although sustainability is a project priority, the limited training resources available in the Regions makes it unlikely that they will continue supporting ARH training at the same level that Youth.Now is doing, after the project ends. · The ASHE “VIBES” RH/ sexuality training being pilot tested through the Ministry of Education schools has been positively received by students, guidance counselors and MOE officials. The MOE and USAID have confirmed that the ASHE curriculum will be integrated with the existing Family Life Education curriculum and implemented in MOE secondary schools beginning in 2004, despite concerns over availability and effectiveness of guidance counselors to implement it. b. Discussion IN keeping with the three-pronged BCC strategy described above (C.), Youth.Now’s training strategy has included activities to1) increase knowledge and skills of providers who care for adolescents, 2) improve skills of adolescents themselves and 3) strengthen efforts that raise ARH public awareness and link Youth.now activities with those of the MOH. In the provider area, Youth.Now focused its in￾service training efforts on training providers from designated project sites in the 9 parishes, though primary focus to date has been on two parishes. In collaboration with JHPIEGO, Youth Now focused pre-service training efforts on training and technical assistance to 20 nursing tutors at 5 nursing schools for development of an ARH curriculum. In the second area, Youth.Now has coordinated training of peer educators through its Conscious Vibes Peer Camp and other peer training interventions in the target parishes. Another USAID effort directed towards improving adolescents’ RH knowledge and skills is an innovative life skills/ sexuality education curriculum being implemented in MOH schools through FHI and ASHE. The third area is being addressed primarily through the communication efforts discussed in section B. of this chapter. However, training has also been directed at specific influential groups of gatekeepers and opinion leaders from the 9 targeted parishes. This training, which is co-ordinated through MSCI and Youth.now, is directed at five gatekeeper groups. The strategy has been for the trained gatekeeper facilitators to serve as multipliers who will continue to educate and raise awareness about ARH among other members of their communities, groups and networks (See Annex E Summary Training Chart). Curriculum Development Workshops and Training of Trainers Prior to establishing youth friendly service sites in the first two parishes of St. Elizabeth and Clarendon, Youth.Now conducted pre-launch community advocacy meetings with leaders from each of five groups (Peers, health providers, male involvement leaders, parents leaders and religious leaders). These were followed by curriculum development workshops (CDWs) with key representatives from each group. 30 After the curriculum development workshops, MSCI followed up with three-day TOT training workshops for parent leaders, religious leaders and male involvement leaders based on the priorities gleaned from the curriculum workshops and advocacy meetings. Other training activities included a field trip to MSCI in New York for peer leaders. During 2002 four draft curricula were finalized and distributed. These are 1) Jamaica Adolescent Sexual and Reproductive Health Trainers Manual and Resource Book for Service providers; 2) Christian Family Life Education; 3) Parent Education and 4) Male Reproductive health. A total of 91 individuals were trained as trainers and facilitators between October 2000 and September 2001. These included 21 Service Providers, 19 Pastors/ religious Leaders, 14 Peer Leaders, 21 Parent Leaders and 16 Male Involvement Leaders. After these initial CDWs and training courses, the graduate trainers and facilitators conducted education workshops and other ASRH activities within their respective groups. According to evaluation interviews, many worked with group members from other organizations to conduct training and Youth.Now staff often assisted with the spin off training. The trained facilitators relied heavily on the curricula in the planning and organization of their respective training activities. This basic training of trainers and facilitators was originally intended to take place over a five-year period but was actually completed in 12 months. Because the MSCI trainers were conducting training in country so often during this period, it was possible for them to follow-up with previous course graduates and offer assistance in their spin off training events so that momentum wasn’t lost. Because so much emphasis has been placed on the multiplier effect of the training, which is designed to ensure support for ARH at the local level as well as it’s sustainability, the project decided to assess the post￾training activities of the original TOT participants through a mail survey. To date 71 out of 89 trainers responded (80% response rate). In addition the surveys revealed that 65 had conducted 222 ARH training-related activities since completing their training. The greatest numbers of activities were by service providers (90), pastors (48) and parents (46). The results are being fed into a recently designed training database, which has been created in order to share training resources and methodologies with key partners and others. Evaluators noted in interviews with NGOs such as the Jamaica Foundation for Children that resources from the database are already being used. For example, the JFC runs a hotline and Youth.Now has provided them with a list of resource people (from the database and from a provider service directory) in different geographical areas and organizations that can be contacted by concerned adolescents looking for youth friendly counseling and services. Formation and Training of “Master Trainers” In July and November of 2002, Youth.Now and MSCI conducted follow-up “master” training for 34 of the 91 trainers and facilitators trained during the first phase of training. These second tier trainers will expand training networks within their own gatekeeper groups and organizations through on-going full scale ARH training activities. It is not mentioned in the training strategy how the impact of the Master trainer activities will be evaluated during the rest of the project period. However, discussions with MSCI trainers and Youth.Now staff revealed that as the project has expanded, there has been an increasing need for trainers to support development of new ASRH activities in the project parishes. Although the project has been using the original TOT trainers and facilitators to support training efforts, there is an increasing need for more skilled trainers (master trainers) who can independently continue the training on their own. MSCI staff expressed a need for continued development of Master trainers so that a critical mass of people from different organizations can respond to the burgeoning number of ARH training needs. 31 Lower Level Results Indicators The two lower level training results that deal with training are: 1) IR3.1.2.1 Improved knowledge and skills in RH and HIV/AIDS/STI of service providers and 2) IR3.1.2.1.1 Strengthened capacity to provide ARH. The indicator for the improved knowledge and skills of service providers result is the number and percentage of trainees who apply skills to subsequent work. Although the project has not yet conducted an assessment of provider use of ARH skills through the intended “mystery client” interviews or exit interviews, there is some information on this from an assessment conducted by Wesley Bernard, M.D. for Youth.Now during 2002. Dr. Bernard found in his assessment of 3-4 clinicians that they were all providing quality ARH care. The second lower level result, of strengthened capacity to provide ARH is to be assessed by calculating the number of individuals trained by ‘master trainers.’ 50 individuals had been targeted for this indicator by the end of year two. Because the project has only recently completed the training of its ‘master trainers’ they have not yet reached this target. However, observation of this training indicates that the trainees have formulated their training plans and will be commencing their activities very soon. It is expected that the project will be meeting and probably surpassing its projected targets in this area during the next year. Community and National Networking Responses to the training have been very positive. The religious community has had a very enthusiastic response indicating a desire for increased knowledge and skills in ASRH. At the end of the original TOT training, the pastors named a co-ordinating body from among themselves that has continued developing coordinated ASRH activities in church communities. They also formed a co￾ordinating body that has continued to work on the curriculum. During evaluation interviews, it was revealed that this network of ASRH pastors continues to meet regularly and many of them support each other in training efforts at their different churches. They clearly felt that this training was meeting a felt need in their communities. One of the church networks, Whole Life Ministries, has piloted an educational activity among 15 schools in the Clarendon project areas using sports coaches at schools to educate and influence youth about ARH. This sports model will now be introduced in the Western Region project sites among 30 schools with coaches and physical education teachers (150 staff). The Pastor co-ordinating committee is working on the development of an “Abstinence Module” that can be used by churches to teach abstinence RH education. One pastor interviewed about this stated that churches preach about how adolescents should practice abstinence but they do not educate them about the issues surrounding sexuality and teach them negotiating and decision-making skills that would help them to remain abstinent. Parent’s groups have developed similar network activities. In fact, the Coalition for Better Parenting co-sponsored the TOT training for parents in 2001. A large number of parenting organizations represented at this workshop has continued to organize training and education activities in their communities and organizations. One model of locating parent education as part of the life-cycle approach to FLE is being tested by the Coalition in a non-project parish and will be reviewed by Youth.Now for possible replication. Youth.Now is also planning to integrate use of the parenting manual developed by ASHE for subsequent parent education interventions. Interviews conducted by mid-term evaluators confirmed that Parent leader trainers and facilitators continue to network among themselves through the Coalition and through Youth.Now. 32 Responses to the peer training have been very positive. The MSCI training of 14 Peer Leaders in New York in 2001 exposed them to a wide variety of experiences and ARH activities that opened their minds to new ideas. In addition, many peers interviewed during the evaluation commented on the “Vibes” Peer Camp training experience conducted during April 2002 at Hampton School. The camp was run by Addiction Alert and the Red Cross and supported by Youth.Now. Although peer leaders interviewed seemed enthusiastic about the training they received, there had been substantial turnover in ARH peers that had moved on from their schools and not been replaced. Numbers were reduced and those that remained were not as active as before, which raises questions about the sustainability of the training. The Men’s groups’ representatives who participated in the MSCI TOT trainer/ facilitator training have also been very active in networking and furthering ARH educational objectives. Representatives from Fathers Inc. have been active in promoting ARH through such on-going activities such as “boys days” in schools, media advertisements and other activities. The representative from the Police Officer’s Association has organized a number of activities around the country through the Police Youth Program network. In implementing these activities he has sought the assistance of other members of the Men’s TOT training group, located in different areas of Jamaica, as well as from the Youth.Now staff and youth leaders. The provider training was also well received. An evaluation of the MSCI TOT training conducted in October 2000 indicates that there were significant gains in knowledge and skills among providers as a result of the training, though there were not significant changes in attitudes. It was concluded that the assets based training approach helped to sensitize participants to the needs of Jamaican adolescents and the importance of providing youth friendly services. Mid-term evaluation interviews with providers who attended the training as well as with Ministry of health officials provided very positive feedback about the training. MOH officials thought that training was one of the strongest aspects of the Youth.Now project and that it had opened the minds of many providers to the needs of adolescents and encouraged them to appreciate the adolescent’s point of view. Though they noted that despite more openness in extending “friendly” reproductive health services to youth, there is still a tendency for providers to recommend abstinence as the method of choice over other forms of contraception. This was verified during evaluation interviews with MOH providers at the project sites who stated that if they counseled an adolescent who was sexually active and requested condoms, their first reaction was to try to convince them to practice abstinence. This focus on abstinence results in missed opportunities to convey safe sex messages and ultimately services to sexually active youth. Although the MSCI training did include Sexual Attitude Reassessment components as well as exercises and discussions about not allowing one’s own beliefs and values to interfere when counseling clients about services, it is clear that more time and participatory learning strategies need to be directed to this area. Similarly, the revised draft ARH curriculum (being supported by Youth.Now and JHPIEGO) that is to be implemented in the nursing schools in 2003, does not appear to be devoting a sufficient amount of time to strengthening ARH counseling skills. Given that this has been identified as a weak area among existing nursing staffs, it would be useful for the project to review it with the curriculum review committee. Documentation from the Youth.Now project indicates that in addition to the MSCI - TOT training for providers and community representatives, Youth.Now has carried out a significant number of ARH 33 training events in the areas of provider education at Youth Friendly Service Sites, peer education and advocacy training. Trainers and facilitators from the MSCI Service Provider TOT and the Peer Leader training have been used to conduct replication of the training to these groups. Between March 2001 and November 2002 a total of 400 individuals have been trained in the Youth.Now training and multiplier training events. As can be seen here, the Youth.Now project has shown an ability to adapt its strategies in order to discover and reach unique groups and networks that deal with or influence the lives of adolescents within the health system and various other communities operating in Jamaican society. Through a variety of strategies, the Youth.Now project is reaching provider and community audiences in the 9 targeted parishes as well as through national networks that include the regional health systems, churches, parents organizations, men’s organizations and nursing schools. The two main training strategies employed by the project include 1) MSCI’s training of 91 trainers who have spun off 222 additional training events through corresponding local and national networks; and 2) Youth.now’s training of 400 service providers, peers and ARH advocates primarily in the active project sites of the SRHA, Hanover and St. Catherine. As noted earlier, FHI and ASHE’s work with the Ministry of Education (MOE) has been very well received thus far. Mid-term evaluation interviews held with ASHE, guidance counselors and MOE officials all indicated an enthusiastic response to the Vibes teaching methodology. MOE officials and guidance counselors interviewed talked about how interested students were in participating in the program’s exercises and group activities. They indicated that whenever a session from Vibes was being taught it drew students from surrounding classes that also wanted to get involved. The Vibes methodology is clearly grabbing the attention of students and according to the preliminary evaluation results, is influencing decisions among boys to practice safer sex. As a result of the 1999-2000 pilot study, the MOE has decided to integrate the Vibes manual and activities into the existing FLE curriculum and implement this integrated curriculum in the public schools beginning in 2004. A concern has been raised by guidance counselors and MOH officials that the guidance counselors often do not have enough time to adequately teach all the sessions in the FLE curriculum. Given the extra physical and creativity demands of implementing the Vibes activities, one wonders if the guidance counselors will feel inspired enough to implement them or prefer to fall back on teaching the more traditional FLE program. To partially address this, the next project will try to widen the training for the group of Vibes instructors to include already trained Vibes facilitators, retired teachers, guidance counselors or school nurses, current education officers and other staff. However, there are still concerns regarding whether there will be sufficient amounts of staff time and interest to adequately implement this creative and demanding curriculum. Additional Issues: In its Training Strategy document, Youth.now has noted a variety of strategies that it is either currently using or plans to use to assess its training effectiveness. These include: pre and post￾tests of training courses; planned intercept surveys and mystery client studies to assess quality of care delivered by trained providers; interviews and observation of providers as part of the YFS certification process; number of independent training events conducted by trained master trainers and facilitators as measured by semi-annual surveys and; the effect of training on attitude of 34 community gatekeepers and policy makers to be assessed using a modified Policy Environment Score (PES) survey. To date, the individual course evaluations and one graduate monitoring survey have been completed. The Policy Environment Survey is in progress. The Strategic Training Plan and project work plans do not include plans/timelines for the other evaluation activities noted here. As the project has expanded its search for avenues to reach adolescents and trained facilitators and trainers have increased their own spin-off activities through local and national networks, the project has begun supporting many new, creative endeavors. Although training has been an important aspect of the project strategy from the beginning, it did not take on an independent identity in the project until the end of year 2 when a separate training co-ordinator was hired who subsequently developed a Strategic Training Plan. While these actions are important and need to be encouraged, it is still unclear how the project intends to organize and co-ordinate all these diverse activities and ultimately feed their results back into the project strategy. During evaluation interviews the Youth.Now staff expressed concern about the lack of MOH or regional training structures that they could work with and could continue to support ARH training after the project ends. There was also frustration over the logistics of training regional health staffs given their complicated schedules. Discussions with the Regional Technical Directors for the Southern and the South East Regions indicated that while they did not have resources to support residential training on the level that Youth.Now is doing, there were other training activities that these offices were supporting at much less costly levels. c. Recommendations · Given the positive results and increasing demand for the project’s ARH training activities, it is recommended that Youth.Now develop strategies for expanding training to broader networks of service providers and communities, keeping in mind cost effective approaches (e.g. non-residential) that can be sustained by partners and gatekeeper groups. · Youth.Now should evaluate current counseling skills training strategies and develop approaches that allow counselors to recognize that adolescent’s RH needs differ according to their sexual experience and other characteristics. Then trainees should be provided with counseling skills and values clarification training that will allow them to separate themselves from personal values and opinions and respond to the adolescent client’s needs based on impartial listening to their requests for RH information and/or services. · Youth.Now should review the counseling training sections of the draft ARH module with the nursing curriculum revision committee and if needed, pilot test it with nursing students to determine if training time is sufficient and materials are comprehensive enough for students to adequately learn ARH counseling techniques. · While current evaluation efforts are useful in providing feedback about the quality of training courses and the multiplier effect of the training, it would be beneficial to have timely information regarding the quality of provider ARH service delivery in order to assess training effectiveness. The project should proceed with planned intercept surveys and mystery client studies as soon as 35 possible. Issues such as the need for more counseling/ values clarification training could be detected through this method and fed back into the training strategy. · The project should include above noted evaluation steps and dates in the Strategic Training Plan Workplan. · In order to support alignment of training activities and results with the project strategy, the project should develop a phased training plan with specific target activities, including numbers of activities and persons to be reached, phased timelines and lead actors (Youth.now staff, facilitators, master trainers etc.) Results of phased training should be fed back to the project and integrated into future activities. · Youth.now should work with the regional Health Authorities to develop a low cost ARH training strategy that can be implemented and sustained by the RHAs. The project should attempt to draw on the training networks being established by TOT graduate trainers in addition to RHA resources. They should also explore alternative training programming such as implementation of short 1-hour updates or refresher-training sessions that can be inserted in busy clinic schedules. · USAID, FHI and ASHE should closely monitor the implementation of the Vibes curriculum and work with the MOE to motivate alternative staff, class periods or other mechanisms to ensure that the Vibes activities and teaching are completely integrated and taught in the public schools. Given the excellent work performed by ASHE in the Vibes approach to FLE, USAID should continue to selectively call on their technical services as needed, taking care to avoid open-ended commitments to meet ongoing operating costs of the organization. 2. Behavior Change Intervention/Behavior Change Communication (BCI/BCC) a.Findings · Key elements are in place at Youth.Now for an effective behaviorally-focused communication program which, with some adjustment of its current efforts, can have greater behavioral impact · However, the BCC Component, in its design, lacks both a sharp, precise focus on expected Behavioral Outcomes and concomitant behaviorally focused communication strategies with better behaviorally linked results indicators. · As a consequence, the BCC effort falters but can be nicely enhanced if the faltering spots were to be better braced with more substantial behaviorally focused initiatives. b. Discussion The most important key elements Youth.now has in place for an effective, behaviorally-focused communications program include: a highly competent implementation capacity; recent 36 successful experience with its advertising campaign; considerable capacity for rolling-out training plans for training a large network of community partners (individuals and institutions); a competent partner for operating a Hotline Information Service; a television series in the making with potential for behavioral impact; and good visibility for Youth.now and ARH issues generated by its public relations (PR) efforts. However, current efforts falter due to: inadequate bonding between the advertising campaign and strategically planned community-level action pressing similar behavioral themes; the absence of large network of “sexually empathic” staff and volunteers in the field who can engage young people in more than an “abstinence only” mode of interaction; insufficient behaviorally focused “advocacy communication” linked to policy behaviors; a public relations strategy not sufficiently focused on behavioral outcomes; targeted community interventions lacking with too limited parish-wide outreach and impact; no marketing communication for ARH services; no significant use of the school system for extensive ARH information dissemination; and no standardized package of ARH information for mass distribution in the targeted parishes. The design of the Program for SO #3 and its Youth.Now project component consists of activities described as “Behavior Change Communication interventions” directed at an IR focused on improved knowledge and skills of adolescents rather than specific, precise behavioral outcomes. The specification of “skills” (which would suggest some behavioral outcomes) is absent in the USAID Health Strategy 2000-2004, the Cooperative Agreement with TFG and partners, the Results Monitoring and Reporting Plan, and Youth.now’s progress reports. In fact, when one looks at the indicators in the Results and Monitoring Plan or the SO #3 results Framework, the BCC component seems restricted primarily to indicators of knowledge of HIV/AIDS transmission, message recall, increased knowledge of the menstrual cycle and conception, methods of contraception, and symptoms of STI. There is some reference to measuring change in community attitudes and proportion of adolescents knowing where to go for ARH services. But on the whole there is very little related to contributing more substantially to specific behavioral results. Even if one accepts this “knowledge” orientation to the indicators, there is very little data available to provide any indication of progress to date, except for the message re-call indicator for the advertising campaign. One could argue that increasing knowledge is an essential step towards behavioral impact. But 50 years of health communication experience would suggest that while improved knowledge is often (though not always) a necessary basis for behavioral responses, it does not provide an adequate and sufficient platform for behavioral engagement. One needs to go beyond knowledge and focus on those other factors which impinge on behavioral decision-making and action, and which are amenable to a communication intervention. For example, if a perception of condom use is linked to loss of pleasure in sex, then a communication intervention needs to address this. One would normally expect that a BCC strategy would have a sharp and precise focus on discrete behavioral outcomes, which then drive the design and execution of the BCC effort. In this project there is no explicit, up-front dedication to specific, precise behavioral outcomes. This is not avoided altogether. But even when addressed in the Work Plan and Implementation Plan for the Project (1999-2001) (with a very instructive conceptual framework attached), the reference is couched in vague terms such as “promote healthy behaviors”, “promote safer sex”, 37 “develop an enabling environment”). There is mention of behaviors such as promote condom use and use of contraceptives, but these lack specificity and precision. What is missing is a behaviorally focused communication strategy, dedicated to specific, precise behavioral outcomes. And each behavioral outcome may very well need its own supportive communication strategy. An all-purpose communication strategy is very unlikely to be effective with respect to all behavioral expectations. For example, the behavior to come into an ARH site is quite different from containing and delaying the powerful urge to have sex for the first time. There was supposed to be a “BCI Strategy” developed by November 2000 but this did not seem to materialize. There is a BCI Strategy Outline but this is pretty much devoid of a sharp, behavioral focus. The advertising agency partner, Dunlop Corbin Communication (DCC), did map out an advertising strategy but this is only one component of a behaviorally focused communication strategy. Training and sexual empathy: Youth.now now has a core of master trainers covering five groups of community-based outreach cadres: faith-based leaders, youth/peer leaders, leadership of parents’ groups, men, and service providers. These master trainers and those subsequently trained by them have the potential to become potent “personal communication agents” for ARH. “Personal communication agents” are persons who have the knowledge, skills and empathy needed to interact and communicate effectively health with youth at a personal level on issues of sexuality and reproductive health. Youth.Now has created this capacity for developing a substantial core of these potential ARH communication agents for ARH outreach at the community level in a large, reachable network of community partners. This now needs to be advanced in a more massive and cost-effective way in the target parishes rather than the current ad hoc approach. In doing so Youth.Now should integrate within its current clinical view of sexuality a better emotional and cerebral grasp of the intense sexual passion, exuberance and urgency which drive sexual behavior among the young. It is clear, for example, that a dominant constraint in engaging adolescents about their sexual behavior has to do with personal discomfort about sex and adolescents in sex or contemplating sex. As commented on elsewhere, the first “preachy” line of “counseling” offered by most service providers and peer leaders is that of abstinence, loaded with what often comes off as an abhorrence of sex. The training provided for service provides does deal with sexuality but does so in a somewhat clinical fashion. About 10% of training time is spent on this clinical perspective of sexuality, with a passing reference that the sexual urge in adolescents is normal. What needs to be added is that this urge is often overwhelming and intense in the minds of adolescents. To talk about abstinence is to talk about a sexually charged behavior. More acute sexual sensibility which needs to be integrated in the training so that “personal communication agents” being trained, be they nurses, peer counselors, community health aides, or dental auxiliaries, will be better able to deal comfortably with adolescent sexual behavior and protective options ranging from abstinence to protected sex Advertising and mass media: Youth.now’s communication partner, DCC, has provided strategic support in publicly positioning Youth.now as a MOH program dedicated to adolescent ARH through selected media presence and judiciously selected public relations activities. Youth.now’s own outreach efforts have also contributed to this. 38 DCC has implemented a striking Phase I advertising campaign (using radio, television, the press, cinema, posters and billboards), nicely segmented, with catchy phrasings and high production value, strategic media scheduling, and with the creative input of young people themselves. Despite the absence of explicit behavioral objectives in the overall Project design or its BCI/BCC component, the advertising campaign emerged with a sharp behavioral focus on abstinence (sexual delay). This has had substantial resonance, it would appear, with key segments of the population. The messages became a bit more muddled, however, in the tail end of the campaign as it pushed both the abstinence (sexual delay) and consistent condom use themes. Promoting what would appear to be inconsistent, even conflicting behavioral messages can be tricky. The consistent condom use theme, however, reinforces well the on-going HIV/AIDS campaign. The impact of the campaign is seen as significant in terms of high re-call and expressed behavioral intentions. But the sampling methodology (an intercept quota sample) of the impact evaluation raises questions about the generalization of the results to the target populations. Even so, it seems that the campaign clearly registered its messages among the target audiences. Given the demonstrated capability for executing a mass advertising campaign and the experience of this Phase I which puts dramatically on the public agenda issues of sexual delay and condom use, future advertising campaigns can play an even more dynamic role in contributing to behavioral impact by linking the campaign closely to synchronized, same message dissemination at the community level by the project’s emerging core of “personal communication agents”. In Phase II it would be useful to re-consider the use of some media, such as posters, billboards, bus sides and bus shelters, which did not seem to be powerful means of delivering the messages. The posters, for example, seem to end up in guidance counselors’ rooms and inside clinics but not much in public community space. One wonders whether it might be possible to do a run of cheap posters as done in election campaigns, with massive community placement. Their short public life can be balanced by the strategic “postering” in relation to a radio-TV-press flight of advertising. In addition, a Phase II advertising campaign may wish to be more directly and personally engaging with target audience members with less emphasis on jingles and dramatic skits and more emphasis on warm, personal sensitive engagement. A Phase II campaign may also wish to address more directly the issue of consistent condom use. In this regard, it may need to address what anecdotally is a primary reason for inconsistent use: loss of pleasure, as perceived by both young men and women. In addition, the campaign should also engage sexually active adolescents in considering a wider range of contraceptive options. One would also urge the development of a new theme associated with consistent condom use or other contraceptive choices, and separated from the popular sexual delay theme of “Since Love So Nice, Wait till it is right”. Perhaps an extension of this theme? “Since love so nice, wait till it is right --- and then do it right---use a condom every time, every time, every time…”. The Youth.Now personal communication outreach potential at the community level together with a mass media advertising competence of DCC augurs well for the powerful combination of advertising and personal communication for behavioral results, were these elements to be strategically synchronized in timing, behavioral focus, consistent messages, and in a style of engaged communication as described above. 39 Youth.now’s partner, CARIMAC, is in the process of completing a 10-part half-hour series of magazine programs on ARH for television broadcast. The programs so far produced have spirited hosts and a visually stimulating mix of segments. While the programs cover a broad range of ARH themes and therefore a muddle of behavioral themes, there is substantial potential here for the series to be used in an engaging manner beyond their solitary broadcasts (time for which has been procured at remarkably reasonable costs by DCC), with a live half-hour chat and call-in television show before a live audience, following the broadcasts. The scripts for the series seem to have been finalized and production is expected to be completed by the end of the year. If there is still an opportunity for change, Youth.now and CARIMAC may wish to focus the content of the series on a few specific behavioral outcomes, including one related to promoting the use of a wider range of contraceptives. In addition, the series may wish to include segments, which rely less on instilling fear as a basis for appropriate behavior, and more on the value gained. For example, sexual delay and condom use is presented in relation to dire health consequences. Fear appeals have proved to be not very effective in prompting healthy behaviors. Audiences disengage as a protective device. Perhaps more engaging would be skits depicting a sense of “good feeling” at having delayed sex, or using a condom and feeling good both about protecting oneself and having pleasure. Youth.now has developed a competent and skilled partner in the Jamaica Children’s Foundation, which is a sub-grantee for enhancing and expanding its Friends Hotline counseling services. The Hotline attracts a modest number of callers who can have access to either personal counseling or taped messages on selected topics. The number of calls seems to go up in relation to advertised promotion of the Hotline. This Hotline offers yet one more personal communication opportunity, which has the added advantage of its anonymity and which can be effectively linked to the advertising campaign, as was done in Phase I. Targeted Interventions: As part of its BCI/BCC strategy, Youth.now has embarked on a number of “Targeted Interventions.” One called Pro-STAR Pre-Teen Intervention is an operational research project involving two schools in Lucea, Hanover and a control school. The goal is to reinforce knowledge, life skills such as reasoning, listening and social skills, with story telling used a primary medium for imparting values and beliefs. In relation to ARH behavioral outcome, the goal is to get a large proportion of the students age 10-12 in the two schools to indicate abstinence as their lifestyle choice. This project began in September. While it may be unfair at this point to comment on this project at this stage, one approaches it with great skepticism. It seems to be one more attempt in the 30-year history of family life education in Jamaica to introduce “life skills” exposure to students. Even if the project succeeds in getting a large proportion to cite abstinence as a life style choice, this may mean nothing behaviorally. Intention to act and action are enormously different. And all the difficulties that Family Life Education has presented in the classroom over the past 30 years will also haunt this project were it to be ever scaled-up for introduction across the nine parishes. The other targeted interventions are focused on boys and their male guardians and involve sub￾grants to the Women’s Center and Children First, and preliminary explorations with Fathers’ Inc. Comments on the sub-grants have already been made in this report. In relation to the BCI/BCC 40 program, one would add the following: Unless these initiatives lead to a mass roll-out of “personal communication agents” who can engage young men on a wide scale in considering specific behaviors being suggested, they will remain interesting small scale exercises, with inconsequential impact on ARH. Targeted community interventions are important especially when strategically and behaviorally linked to a national mass media advertising campaign. One would urge for large-scale targeted community interventions in this regard. The nine parishes represent a small enough “market” for such scaled-up community interventions involving 400,000 adolescents and their families. Youth.Now (with its soon-to-be appointed new Communication Officer) and DCC have the creative capacity to conceive of what sort of scaled￾up, parish-wide targeted community intervention would have impact with 400,000 adolescents at modest costs. BCI/BCC at service sites: Youth.Now has a major mission of getting adolescents to avail themselves of ARH services at Youth-Friendly (YF) sites being established under the project. The “Services” section of this report covers some of the deficiencies in this area. From a BCI/BCC perspective, if ARH services are inconvenient or unattractive or unappealing to prospective consumers, then it is both difficult and pointless to promote them. If a YF site offers (as discovered in some locations) services only on the third Wednesday of the month and for just two-three hours and with a potent dose of preachy abstinence being offered, it is perhaps wise not to promote such a site. However, the critical need for ARH service sites remains. Youth.Now may find solutions within the NGO and commercial private sector, in addition to government clinic-based locations, with more appeal to prospective users. As these sites become more available and more attractive to consumers, Youth.Now should consider a vigorous marketing communication program for marketing “branded” ARH services in the public, private and NGO sector and for prompting greater use of these services. This may involve use of mass media advertising even if the services for the moment are only in just 9 parishes. But even more important is the community￾based marketing communication, which will need to be carried out. For example, any appealing ARH service site should carry whatever logo/brand name is developed for this purpose. In addition, impressive point-of-service promotion should be undertaken. Youth.Now and DCC have given thought to this area of action and have developed possible logos and signage but have not ventured further, partially because there isn’t much to “market”, and there is a bit of caution about branding and so appearing to separate the project from the MOH. Advocacy communication: An important component of the Youth.Now project relates to policy development and implementation. There is a particular policy issue which, from moment to moment, is either off or on the table. It has to do with the age of consent and parental rights so as to allow provision of clinical ARH services to adolescents under age 15. To get a policy adopted and implemented calls for certain people to take certain actions, and here emerges a behavioral focus. In the policy area, communication strategies dedicated to prompting action are described as “Advocacy Communication”. Youth.Now has a modest effort in this field. Most of the current advocacy efforts relate to generating community and public support for ARH in a general sort of way. There is the excellent beginning of a Speaker’s Bureau, modest presence in the media by way of radio-television interviews, newspaper coverage of events, exhibition booths at health fairs etc. This is all good public relations, public positioning of Youth.Now and ARH themes. 41 But it lacks the sharp behavioral focus for policy action. If the policy document on age of consent is temporarily lost amidst all else on the Attorney General’s desk, DCC and Youth.Now should work towards an Advocacy Strategy which has as its behavioral goal getting that document to the top of the pile. Public Relations: Related to the above is another gap which can be easily remedied: a marketing public relations effort linked to specific behavioral outcomes. DCC and Youth.Now have done much to give visibility and prominence to the project and ARH themes. But with a bit more tweaking, a powerful non-advertising marketing public relations campaign can be mounted to lend support to specific, precise behavioral outcomes. By way of a minor example, one such activity in a marketing PR effort focused on the behavior of consistent condom use could have a radio call-in show with planted calls to begin with which focus on the perception of “loss of pleasure” in condom use. Both men and women can comment on this from a variety of perspectives, including that of the cost vs. value in consistent condom use: a little loss of pleasure vs. the value of protection and how much at risk does one feel. Newspaper feature articles and TV chats shows could also focus on this theme, plus community meetings involving the male segment of the “personal communication agents.” As mentioned earlier, DCC has provided effective public relations support in putting both the project and ARH on the public agenda; and there are plans for additional PR activities along similar lines, using popular media programs, including chat shows and call-in shows. What is being suggested is that future public relations efforts take on a much more rigorous behavioral focus and be strategically designed in relation to agreed-upon behavioral outcomes, rather than just build generic support for ARH. Informational materials in schools and at service sites: Schools currently appear to be a major source of ARH information for adolescents, even though ARH information is haphazardly offered in the school system. In addition, roughly 80-90% of children 10-17 are in schools. While it seems difficult to get an organized delivery of a family life education curriculum (such as the one which now exists) to each child, the school system remains a powerful channel for at least the dissemination of standard information materials on ARH to each child 10-17 years of age. Even if inspired teachers and guidance counselors do not use a dissemination opportunity for in-class discussion, the children will provoke their own internal discussions and chats with other children based on their reading of ARH information. Perhaps a standard ARH workbook could be cheaply printed in mass quantities (100,000 for the age group 10-12, and 200,000 for the age group 13-17) and distributed to each child in the appropriate age group in the nine targeted parishes. At the same time teachers and guidance counselors could be encouraged at their own convenience to foster discussion of the issues raised in the workbook. There is also a dearth of information materials in the project for easy, wide scale distribution to individuals in communities and at service sites. Youth.now has carried out an inventory of selected ARH materials currently available. In a joint purchasing and printing process with GTZ, USAID and the MOH, the project has procured small numbers of selected pamphlets and brochures: 7,000 “Sex, Am I Ready”; 3000 “Common Sexually Transmitted Diseases”; 2,000 R U Safe; 5,000 “On Becoming a Sexual Person”. These quantities may be fine for the numbers of “personal communication agents” being created but they are rather tiny in relation to a 42 population of 400,000 individuals age 10-19 in the nine targeted parishes. Ideally, each of the 400,000 individuals should have some kind of ARH overview literature in their hands, plus behaviorally specific pieces related to the behavioral themes being pressed. In trying to do this in a cost-effective manner, one may need to sacrifice a bit of quality and the desire for full-color materials in favor of mass quantities for mass distribution. c. Recommendations · Sharpen the behavioral focus of the project and design a new behaviorally focused Strategic Communication Program with better behaviorally linked results indicators. · Establish a large network of sexually-empathic “personal communication agents” for extensive community outreach and targeted community interventions through structured training plans using the team of master trainers and associate trainers so far developed in the five “market” categories: faith-based leaders, peer leaders, service providers, male leaders, parent leaders. · Conduct a Phase II Advertising Campaign better linked to scheduled/planned community outreach of the “personal communication agents” and wide scale targeted community interventions, with a modified selection of media interventions and in a more personally engaging format, and with greater behavioral attention to consistent condom use, and use of a wider array of contraceptive methods for the older adolescent market segments. · Re-examine the content themes of the CARIMAC “YOW” TV Series from the perspective of adding more of a “positive modeling” dimension rather than motivating behavior on the basis of fear appeals; explore broadcasting the series followed by a half-hour live chat/call-in television show before a live audience to foster more viewer engagement. D. National Policies and Guidelines 1. Findings a) Youth.now and the Policy II Project have made significant contributions to the ARH policy agenda in Jamaica. While no new national policies and guidelines have yet been implemented as a result of these efforts, it is expected that a new National Youth Policy and Strategic Plan will be adopted and implemented over the next two years. b) The Youth.now project was instrumental in the formation and ongoing support of the Policy Working Group (PWG). However, the work of the PWG has been uneven and not fully effective. c) Youth.now currently lacks a clear strategic focus for its work in the policy area. 2. Discussion 43 Though not included in the ARH program description of original Strategic Objective Grant Agreement (SOAG), the policy component was added in May of 1999 in recognition of the importance of supportive national policies in the creation of a positive enabling environment for ARH activities. Subsequently, work at the national policy level was included in the contract with the Futures Group for the Youth.now project, where it is described as assisting the MOH and NFPB as needed with technical assistance, along with workshops and seminars to promote policy reform. In addition, and independently of Youth.now, USAID has provided funding directly to the POLICY II project for technical assistance to the MOH and to the NCYD. POLICY II support to the MOH is directed at strengthening strategic planning for reproductive health at the regional and parish levels, while its support for the NCYD is directed at assistance in formulation of a National Youth Policy and Strategic Plan. By all accounts, the performance of the POLICY II Project in both key areas has been excellent. POLICY II sponsored strategic planning workshops in all four MOH regions, which produced strategic plans for reproductive health activities in each region. The process is now underway to complete parish-level strategic plans based on the regional plans. The decentralized MOH system places responsibility for the development, management and funding of health care services with its regional health authorities. Therefore, the design and implementation of such plans has the potential to create a strong institutional basis of support for RH activities through MOH programs and facilities. This is especially true if MOH decision-makers are committed to executing the plans, which seems to be the case from interviews conducted by the team with senior MOH regional officers. POLICY II followed up the strategic planning workshops with regional policy advocacy workshops. These workshops also generated a considerable amount of enthusiasm among MOH attendees. Current plans include continuing technical support to the MOH regions and parishes for work in policy planning and advocacy. The South East Region has also requested POLICY II assistance in helping it design more effective systems of program budgeting. POLICY II will require additional financial support from USAID/Jamaica to continue these worthwhile activities. In addition to its work with the MOH, POLICY II has also worked closely with the NCYD to assist that agency draft a comprehensive National Youth Policy for Jamaica. It is expected that the Youth Policy will be finalized and submitted to Cabinet for approval early next year. The new Policy covers many areas important to youth development, such as education, training, employment and living environments. While the health section of the Policy is brief, it does address specific reproductive health needs of adolescents and endorses the development of new programs to address them. The Policy also includes provision for development of a National Strategic Plan for Youth Development, under which specific government agency implementation plans and programs will be developed and monitored by the NCYD. The NCYD is very pleased with the quality of POLICY II’s contribution to this effort and has requested that its support continue through the development of the important implementation phase of the process. While POLICY II efforts to date merit high praise, the picture is more mixed with respect to Youth.now’s performance in the policy area. Perhaps Youth.now’s most important contribution to date has been the development and dissemination of several important policy research documents including a literature review of ARH policy research, and analysis of the legal, 44 regulatory and policy environment for ARH in Jamaica and an analysis of steps in passing a law or policy in Jamaica. Such studies comprise an important basis for development of a plan and approach to ARH policy reform. Youth.now has also been instrumental in the formation and support for the ARH Policy Working Group (PWG), a multi-sectoral body jointly chaired by the PIOJ and the MOH. The task of the PWG is to guide the development of appropriate ARH policies and guidelines. This far, two sets of guidelines have been developed under the aegis of the PWG, one addressing the problem of current legislation prohibiting the provision of contraceptives to minors under the age of 16 (below the age of consent) without parental consent, and the other dealing with the treatment of adolescents for STIs, including HIV/AIDS. The prospects for adoption of the guidelines are uncertain. The new guidelines allowing the provision of contraceptive services to minors who are below the age of consent were submitted to Cabinet by the MOH last year, but were withdrawn because of their political sensitivity. The office of the Attorney General is currently reviewing them. Unfortunately, the consensus among senior MOH officials is that neither the Ministry nor the Cabinet has the political will necessary to adopt the new guidelines. Current MOH guidelines for RH allow for the provision of contraceptives to minors below the age of consent without parental consent under the so-called Gillich Law, but lack of awareness of this provision among many MOH staff and uncertainty as to its legal standing continues to constitute a barrier to services. The drafting of new guidelines for treatment of STIs is less advanced and has been held up pending finalization of the related Child Care and Protection Bill. Although the broad multi-sectoral composition of the PWG makes it an ideal forum for ARH policy development, it has not lived up to its full potential. It has been nearly a year since its last meeting and none are currently scheduled. Nor does there seem to be a clearly defined set of priorities for future action by the PWG. On the other hand, the NFPB, which did the actual drafting of the new guidelines for contraceptive services to minors, has expanded its policy division and the NCYD has begun to play a more aggressive role in the youth policy arena. The PWG, with Youth.now assistance, needs to reexamine its role vis a vis continuing priorities for ARH policy reform at the national level, especially in light of the heightened roles being played by NFPB and the NCYD in this area. The October 1 2001 to September 30, 2002 Youth.now work plan identifies a number of priorities for IR 3.1.3 (Policy) for the year, including support for the PWG in preparing policy guidelines for MOH approval, conducting outreach and capacity building with key policy makers and stakeholders in support of ARH, support for development of an ARH policy and for the National Youth Policy and planning for the next round of the Policy Environment Score (PES) survey. On balance, very little seems to have been accomplished during this period. As previously mentioned, the PWG has been virtually defunct for most of the last year. Outreach and capacity building among key policy makers has not taken place, partly because of staff turnover in the Youth.now policy specialist position. A new full time policy specialist (Advocacy Coordinator) has recently been hired which should result in more attention given to this area in the future. However, it is important that the role of the new advocacy coordinator, which currently seems to be oriented more towards support for behavior change activities at the community level, be geared primarily towards policy advocacy activities among key decision makers and stakeholders. Plans for the development of an ARH policy have been put on hold in 45 view of the fact that similar issues are being addressed through the National Youth Policy. Finally, the PES, which is conducted with support from the POLICY II Project, is scheduled to take place during the first quarter of 2003. The PES is the principal indicator identified in the Results Framework for measuring achievement of the Policy IR. It consists of a composite of factors such as political support, policy formation, resources, programs, etc. considered key to measuring the degree to which Jamaica supports the reproductive health of the population. The last PES survey was conducted two years ago in November/December of 2000. The results of that survey showed a modest increase of 5.7% in the composite score of the adolescent element over the 1999 baseline. While this was encouraging, it should be noted that the overall score of the adolescent element (59.7) of the PES was significantly lower than other elements such as family planning (71) and STDs/AIDS (74.1). It is anyone’s guess how the forthcoming PES will turn out, though it is likely there will be a modest improvement due to increased program effort over the last two years. In addition to the PES, the Youth.now RMRP contains lower-level indicators for measuring performance in the Policy IR. Here again, performance is mixed. The target of at least one new policy or guideline submitted to the MOH for approval has been met with the proposed new guideline relating to the age of consent for contraceptive services to minors, though the prospects for actual approval seem remote. Another target is revised curricula in educational institutions reflecting an ARH focus. This target has been partially met, in that a revised curriculum has been developed with assistance from JHPIEGO for use in nursing schools, which is now being considered for adoption. This indicator more properly belongs under IR 2 Improved Knowledge and Skills (Training). Finally, there is an indicator, which consists of the adoption of a consensus definition of adolescent reproductive health. Since the PWG has adopted such a definition, this indicator appears to have been met. However, one has to question the utility of such a definition, given the way in which it is formulated. The definition, for example, includes the “emotional” well being of adolescents along with their physical well being as well as “freedom from sexual violence and coercion”. While it is certainly desirable that youth will be emotionally fit, it is difficult to imagine how ARH programs can or should be designed to achieve this end. On the other hand sexual violence and coercion by adults does play a direct and important role in adolescent sexual behavior and outcomes, and should be more fully addressed in ARH programs. 3. Recommendations a) USAID/J should provide additional funding for the POLICY II project to ensure that: a) the NCYD continues to have the technical and program support it needs to finalize the National Youth Policy and to develop its follow-on National Strategic Plan for adolescents and, b) technical assistance and training continues with the MOH Regional Health Administrations in strategic planning and advocacy for ARH at regional and parish levels. 46 b) USAID should encourage the Minister of Health to request that the Cabinet take positive action on proposed new guidelines related to the provision of contraceptive services to minors who are below the age if consent. The Minister should also be encouraged to issue a clear statement of support for current MOH guidelines allowing for such services under the Gillich Law. c) Youth.now should assist the PWG to redefine its role and action priorities for ARH policy reform. In addition, it should redefine its own role vis a vis the PWG, and other key GOJ agencies in support of national ARH policy and advocacy. IV. PROJECT MANAGEMENT A. Findings: 1) Overall, The Futures Group has performed well in managing the Youth.now project in Jamaica. 2) The absence of the MOH “focal point” for the project has adversely affected communications and coordination of project activities within the MOH. B. Discussion: The Futures Group has performed well in managing the Youth.now project in Jamaica. Experienced and capable personnel, both at the home office and in Jamaica, staff the project. Staff turnover has not been a problem, with the exception of the part-time Policy Specialist position, which has been recently filled and converted to a full time Advocacy Coordinator position. The recent departure of the IEC Specialist will require that this critical position be filled as soon as possible. The quality of project performance would also benefit with the addition of staff with expertise in the design and management of RH service delivery programs at scale, and HIV/AIDS. Project funds appear to be managed properly and prudently by both the corporate office of Futures and locally. The management and coordination of short-term technical assistance and Futures partners MSCI and DCC is performed efficiently. There is room for improvement of coordination of effort with some CAs such as Family Health International (FHI), though coordination with others such as the Policy II Project is excellent. Youth.now has established close and effective working relationships with USAID/Jamaica and with its Jamaican partners in the public and NGO sectors. The evaluation team heard frequent expressions of praise for Youth.now from MOH officials at national, regional and local levels for the quality and extensiveness of their collaboration through numerous training activities, project committees, working groups, ad hoc meetings and other forms of intercourse. Nevertheless, the departure of the MOH “focal point” for the project in June 2001 has left a gap in communication and coordination within the Ministry. Currently, it is difficult for the overburdened head of the Health Promotion and Protection Division to fully engage the Ministry in project activities, as was the case when she was assisted in this task by the focal point. Funding and bureaucratic 47 constraints have made it difficult for the Ministry to fill this position, though the team was advised that the Ministry would now make renewed efforts to do so. It was also suggested to the Team that Youth.now’s work would be facilitated at the MOH field level were the project office to be moved to the Family Health Division from its current location in Health Promotion and Protection. Though a good argument can be made for such a move, the Team is not persuaded that it would be advisable at this relatively late stage of the project, given the already well￾established working relationships that exist between the project and Health Promotion and Protection, not to mention the fact that the nature of most project interventions are more appropriately educational and promotional than they are clinical. Notwithstanding some bumps along the way, such as Youth.now’s occasional failure to acknowledge its status as a USAID project in its communications with the public, the quality of The Futures Group’s working relationship with USAID/Jamaica is excellent. Futures has been highly responsive to USAIDS frequent requests for information and suggestions for modification of project activities contained in annual work plans. The USAID ARH project officer meets on a regular bi-weekly basis with Youth.now project staff to monitor activities and to help overcome bureaucratic and other obstacles to performance. In addition to these and other frequent informal ad hoc meetings and consultations between USAID and project staff, a Project Coordinating Committee meets quarterly and an Extended SO Team meets annually to help guide implementation of the project. Finally, the project has complied fully with the terms of its contract with USAID and last year was judged to have performed at a “high standard” by the Award Fee Committee in accordance with evaluation criteria, which included field implementation, administrative performance, mission support activities and cost control deliverables. C. Recommendation The MOH should fill the position of “focal point” for the project as soon as possible. V. PROGRAM SUSTAINABILITY By “program sustainability” we mean the continuation, in whole or in part, of an ongoing ARH program, project or activity without the need for ongoing external technical and financial assistance. Sustainability necessarily implies that the “sustaining entity” (e.g. MOH or NGO) has the commitment, ability and resources necessary to ensure continuation of the program, project or activity in question. 1) Findings a) While it is still too early to tell if much of the ARH program will produce sustainable results, it clear that the implementation of the entire package of Youth,now interventions as presently configured is not sustainable. b) Some ARH program activities such as training do not appear to be sustainable as currently implemented (e.g. relatively high cost residential approach), though they may inspire the development of low cost alternatives. 48 c) Other strategies and approaches, e.g. involvement of faith based organizations (FBOs), do appear to have the potential to produce sustainable results. d) Institutional constraints may retard the integration and expansion of ARH interventions and activities within the MOH and the NGO community. e) Youth.now lacks systems to adequately identify and roll out successful ARH interventions. 2) Discussion The issue of sustainability is problematic. It requires not only a precise definition and identification of the program element or activity one wishes to sustain, but the ability, resources and commitment on the part of the sustaining agency to carry it out. In some cases, it may also require additional donor resources for a transitional period of “dissemination and roll out” in order to meet start up costs and to allow the new activity time to fully take hold. Timing is also important, since premature termination of external support may be fatal, while prolonging “hand off” may lead to complacency and undermine the necessary commitment of the sustaining agency. Although it is still too early to tell if much of the ARH program will produce sustainable results, it is important for USAID and implementing partners to begin thinking about and planning for sustainability as soon as possible. Moreover, even at this relatively early stage, there is more than enough experience and performance data to be able to make informed judgments on sustainability of key elements of the program. For example, it is clear that the ARH program as presently constituted, consisting of the activities of Youth.now and other USAID funded CA activities, is not sustainable and will last only as long as USAID continues funding for it. The same may be said of the Youth.now project. There is no single GOJ ministry or agency that has the broad mandate to encompass the full scope of Youth.now activities and interventions, even in the very unlikely event it had the funds to do so. What then, about the prospects of the sustainability of discrete key activities the CAs and Youth.now have been undertaking? First, with respect to activities of the CAs, it is important to keep in mind that much of their work was designed to fill a discrete need in support of a larger effort and were never intended to be sustainable in and of themselves. This is true, for example, of the work done by MSCI, JHPIEGO and DCC as well as AED on the CHANGE project and special studies done through FHI on the Behavioral Surveillance Survey and CMS on the Adolescent Condom Survey. On the other hand, it is hoped that technical assistance provided by other CAs to strengthen the performance of already existing systems in the MOH such as installation of a new clinic appointment system and improvements in the Ministry’s health management information system will be sustainable, but it is too early to tell. With respect to other, more far-reaching interventions such as FHI’s work with ASHE and the MOE to install in the Guidance Counseling Division of the MOE a new approach and curriculum for teaching family life and sex education in the public school system, it is likewise too early to tell if this effort will continue beyond next year when funding for the program runs out. The Vibes curriculum developed by ASHE has 49 many enthusiastic supporters in the MOE, which has pledged to continue funding the program through the government budget beginning in 2004. This augurs well for the continuation of some form of the Vibes “edutainment” approach in the future. In addition, the technical assistance and training provided by POLICY II in strategic planning and advocacy at the MOH regional and parish levels, seems to have generated a desire among some regions to continue this process at some level on their own, though it is again too early to tell if this will happen. With respect to Youth.now interventions, the YFS sites in MOH clinics are not producing the desired results in terms of reach and generation of new contraceptive users, and for this reason alone they are probably not sustainable. In addition, there are important constraints inherent in the MOH institutional environment, which make absorption and continuation of new programs and activities by the Ministry very difficult. These include shortages of staff and funding, restrictive policies and practices (especially as they apply to adolescent programs), competing health priorities and lack of commitment and political will necessary to overcome bureaucratic inertia and resistance to change. This is not to say that sustainability of successful ARH activities is impossible in the MOH; it is possible, but it is also difficult. Perhaps the most promising Youth.now candidate interventions for sustainability within the MOH are the provider training activities that have generated so much enthusiasm and acceptance among the MOH staff who have been exposed to the training. It is quite possible that a more sustainable low cost (e.g. non￾residential) approach to such training through regional MOH structures could be developed, and Youth.now should begin working with MOH authorities at this level to develop a plan to incorporate such training interventions in ongoing regional and parish training programs. As discussed previously, Youth.now is working with a variety of Jamaican NGOs to promote ARH in different settings. Unfortunately, as was also noted, although these programs are still relatively new, the results thus far have not been encouraging, both in terms of the small numbers of adolescents reached with counseling and education programs as well as the weak links to reproductive health services. With respect to sustainability, chronic problems of underfunding, limited absorptive capacity and limited reach (i.e. most NGOs are concentrated in a few metropolitan areas of the country), make NGOs, in general, poor candidates for sustainability of program efforts. The example of FAMPLAN is instructive. This venerable organization has been in existence for decades. It has an institutional mandate to promote the expansion of family planning and RH services in Jamaica. Yet, despite support from an ongoing grant from the IPPF for operating expenses, it has not been able to continue, much less expand, the many donor funded programs it has attracted over the years, including various programs directed at adolescents. The situation with other Jamaican NGOs without FAMPLAN’s RH mandate and ongoing external financial support is much direr. It would appear that some potential for program sustainability within the NGO community lies with FBOs and youth serving organizations with large networks (e.g. police youth clubs) who are able to adapt ARH youth friendly approaches to existing programs with few additional costs. Finally, as noted above, it is essential to be able to clearly identify successful interventions and to then take concrete steps to expand and roll them out. However, Youth.now, although it does collect a lot of program performance information, does not currently have in place the necessary evaluation systems which would allow it to clearly identify successful interventions amenable to adaptation and expansion by Jamaican agencies and organizations. Just as importantly, 50 Youth.now lacks a “roll out” plan which would allow it to take the desired intervention, adapt it appropriately, and through the use of technical assistance, dissemination seminars, etc. ensure its transfer to the sustaining organization. 2. Recommendation. Youth.now should institute a process for identifying and rolling out successful ARH program interventions VI. CONCLUSION The overall conclusion of this evaluation is that, while some ARH interventions (e.g. training), seem to be on track towards meeting and perhaps even exceeding expectations, the program as a whole does not appear to be achieving its most important intended results. Specifically, and most importantly, YFS sites at MOH clinics and among NGOs are failing to attract significant numbers of new family planning acceptors among sexually active adolescents. Moreover, there is little evidence that the current YFS approach will produce this result in the future. In addition, linked community outreach efforts in the schools have produced a small cadre of committed youth, but this effort is limited, labor intensive and does not appear to be self-sustaining. Nor does it appear that it is resulting in increased condom use among sexually active in-school youth. ARH BCC and training efforts have been enthusiastically received. However, the effectiveness and sustainability of these efforts are unknown. The emphasis the project is placing on abstinence is an important and innovative intervention that needs to be explicitly addressed in the RF. However, it remains to be seen whether this message will ultimately prove to be effective in delaying sexual initiation among teens. In the meantime, more attention needs to be given to the problem of the abstinence message getting in the way of meeting the contraceptive needs of sexually active teens who have no wish to abstain. Finally, POLICY II’s work with the NCYD and the MOH regions has progressed well, but Youth.now’s work in policy has been more uneven and less promising, and the overall policy environment continues to be less than favorable towards ARH programs and services. Since only two years remain in the current Youth.now contract, now is the time for USAID/Jamaica to take concrete, decisive action to refocus the ARH program. This will not be easy, though a number of important factors bode well for success of such an effort. First and most importantly, the ARH program is blessed with partners at USAID, the MOH and FGI who are talented, knowledgeable, experienced and committed. The fact that the YFS sites are not producing intended results is not in any way an indictment of the TFGI team or the MOH staff, both of whom have worked diligently and creatively to make the effort succeed. Rather, the problem is more one of a flaw in the original program concept and design for the YFS sites, which did not fully recognize the strong resistance teens have to clinic-based RH programs. Moreover, the original design omitted support for other clinical and commercial/private sector efforts directed at teen mothers and older sexually active teens that could have had a more direct impact on increasing contraceptive use. Instead, the Youth.now approach has placed great emphasis on certain interventions that appear to be theoretically sound, but may not have much 51 practical value in meeting the needs of sexually active teens (e.g. efforts to train clergymen to counsel teen to practice abstinence). The foregoing highlights an important issue that needs to be addressed as a matter of priority by USAID and its partners. That issue is the need for USAID, Youth.now and the MOH to reach consensus on the fundamental goals of the project. The evaluation team has assumed from its review of the official documentation and discussions with USAID officials that such goals revolve primarily around developing strategies and interventions aimed at contributing to a reduction in adolescent fertility primarily through increasing contraceptive use among large numbers of sexually active teens at risk of unwanted pregnancy and STIs, including HIV/AIDS. This perspective does not appear to be fully shared by Youth.now, however, who have adopted more of an experimental and indirect approach to reaching more limited numbers of adolescents with a range of information and services that extend well beyond contraceptive services. These perspectives are not necessarily mutually exclusive, but they lead to different approaches and outcomes, especially in the short and medium term. Lack of consensus also creates confusion over what constitutes “success”. Youth.now, for example, may consider it a “success” that they have “reached” groups of teens and influential adults in a defined geographic area with a variety of ARH, life skills and other related (and sometimes unrelated) messages. On the other hand, USAID might consider that same intervention a failure because it has not resulted in a measurable increase in contraceptive use among those sexually active teens. While the Youth.now contract does provide a basis for the “experimental” approach to developing a variety of models for ARH information and services, it seems clear at this point in the project that such an approach is no longer appropriate. This is both because the models are not having much of an impact on increasing contraceptive use and because the necessary systems are not in place to adequately identify, measure and roll out “successful” models and interventions. The foregoing is not to say that a broad approach to reaching adolescents with ARH information and services should be abandoned altogether. There is empirical evidence from other programs and other countries that confirm the strong links that exist between adolescent behavior and the “supporting environment”. A more holistic approach that includes both strong contraceptive information and service elements along with the kinds of broader community BCC, training, policy advocacy and similar interventions, which TFGI/Youth.now is promoting, could work. The basic problem with the current overall ARH program is that it does not focus sharply enough on fertility reduction and contraceptive use as the principal focus and objective of this broader approach. In addition, as has been detailed in the body of this report, the RH service element is very weak and very limited in scope. A sharper segmentation strategy is needed that gives much more emphasis to RH services as part of a holistic approach that also includes more widespread application of appropriately revised (e.g. lower cost, less judgmental) training activities along with more behaviorally-focused BCC efforts. It was noted at the beginning of this report that mounting successful ARH programs anywhere in the world is very difficult. It is especially difficult in Jamaica with its social norms that are characterized by early initiation of sexual activity, yet frown on the use of contraception by the young. Despite these obstacles, strong program efforts in past years directed at adolescents by the MOH and NFPB (e.g. the former “nurse educator” program in public hospitals and clinics together with extensive community outreach and contraceptive distribution programs) were 52 successful in increasing contraceptive use and lowering fertility rates among adolescents. The evaluation team believes that renewed and redirected efforts along the lines of the detailed recommendations in this report will also produce successful outcomes in meeting the reproductive health needs of Jamaica’s youth. Moreover, one would expect that if done in a manner that establishes a firm basis for sustainability, such renewed efforts will have much more profound impact than more limited similar efforts have had in the past. 53 ANNEX A SCOPE OF WORK EVALUATION OF THE JAMAICA ADOLESCENT REPRODUCTIVE HEALTH (ARH) PROJECT 1. Activity to be Evaluated Project Name: The Adolescent Reproductive Health Project Project Number: 532- 0184 Authorized LOP Funding: US$12.0 million PACD: September 30, 2004 2. Background Adolescent reproductive health is a critical area of concern for the Jamaicans under the age of 20 who constitute 20% of the population. Although total fertility rates have declined significantly throughout the island since the 1960s, the rate among females 15-19 years old has increased. Since the 1970s, the proportion of births to mothers under 20 years of age has remained at about 30% of all births. Interrupted and discontinued schooling, a high incidence of pregnancy and related complications, and the lack of social and economic preparation for effecting parenting are some of the main consequences resulting from adolescents’ lack of preparation for healthy reproductive behavior. The Jamaica Adolescent Reproductive Health Project (ARH) supports the Mission’s strategic objective of Improved Reproductive Health of Youth by increasing the use of quality reproductive health (RH) and HIV/STI services and preventive practices. This five-year, US$12.0 million activity is now entering its third year of implementation. It has been designed around the premise that increased use of RH services and participation in youth-friendly RH activities can be achieved by: · increasing access to quality reproductive health and HIV/STI services; · improving knowledge and skills related to reproductive health and HIV/AIDS/STIs; and · supporting reproductive health national policies and guidelines. The lion’s share of ARH activities is being carried out by The Futures Group International (FGI), on behalf of the Ministry of Health. Additional activities are being implemented by a diverse 54 group of international technical assistance firms, local non-governmental organizations (NGOs) and other sectoral ministries which implement programs related to adolescents. This combined effort seeks to increase the use of reproductive health services by young Jamaicans by developing and promoting the use of youth-friendly health care centers and by supporting community-based education, information, and counseling activities. Support is also provided to help improve the policy environment for adolescent reproductive health. TFGI’s work includes offering a comprehensive package of technical assistance and training as well as upgrading supplies and equipment to improve the quality and accessibility of services to youth. In addition TFGI will develop innovative approaches to providing reproductive messages to youth, monitor behavior change and strengthen nursing and medical curriculum as it relates to reproductive health of youth and play a supportive role in the development of national policies and guidelines. Work being carried out by other organizations includes, strategic planning and policy formulation, quality assurance, school-based ARH interventions, research/studies and behavioral surveillance surveys, and training. To date, the key ARH results/interventions have included, establishment of youth friendly sites, development and implementation of a comprehensive mass media campaign, implementation of a training program, production of an ARH learning package, establishment of a small grants program, initiation of a public relations and advocacy component. In addition strategic planning reproductive health workshops have been completed, multi-sectoral approach to youth development has advanced significantly, Quality assurance interventions on MCH and Family Planning standards have been completed, Certification of youth friendly standards and criteria have been pilot tested, Vibes curriculum has begun the initiation in the Ministry of Education Curriculum, and ARH surveys have been disseminated. The ARH project monitors its results by collecting both quantitative and qualitative data from multiple sources. Sources of primary quantitative data include community sample surveys (baseline and endline) collected in target and control communities, exit interviews and mystery client studies, post launch media communication surveys, policy environment score survey, simple service statistics and information monitoring systems at the community and parish levels. Qualitative analyses include key informant interviews conducted in select communities, minutes of advisory group meetings field notes of the Youth.now team. However, data are presented on a general level and do not allow for sufficient analysis to determine those activities that have been particularly successful as well as those that warrant increased scrutiny and modification. The evaluation will examine ARH in light of the interventions that have been developed and their relation to reaching the strategic objective. The results of the evaluation will be used to fine-tune ARH in order to make the best use of program funds and maximize the impact of the ARH program. 55 3. Purpose The overall purpose of this evaluation is to assess the progress and effectiveness of the Jamaica Adolescent Reproductive Health Project (ARH) and to make recommendations for modifications in project design and implementation for the remaining life of the project. The focus of the evaluation will be on examining the major project interventions, their progress and effectiveness, and potential application and replicability within the Jamaican health system and other social and educational sectors. The specific objectives of the evaluation are: · to review the validity of the overall design, strategy and methodology of the project and make recommendations for modifications if deemed necessary; · to assess the effectiveness of project interventions, including an analysis of their potential replicability and expansion to other health and non-health facilities in the system, in terms of their relative contribution and impact on improving the reproductive health of youth; · to assess the managerial and technical performance to date of the project’s technical assistance contractor and grantees, and recommend changes as needed; · to assess the extent to which the institutional environment in the health and NGO sectors are conducive to the incorporation and expansion of successful ARH interventions island-wide. Factors to be addressed include the appropriateness of current applicable health policies and practices, financial capability, physical infrastructure, and political will. · to provide USAID with a comprehensive report on the performance and effectiveness of the ARH project to date, along with recommendations for changes in the design, strategy, and execution of the project needed to ensure successful achievement of project objectives. 4. Statement of Work (SOW) The evaluation team will be responsible for carrying-out a comprehensive assessment of ARH program planning, administrative management, and implementation. The parameters for this effort are described below. 56 A. Assess the performance to date of ARH interventions. ARH is implementing a multiplicity of different, but interrelated, interventions through the work of the Institutional Contractor (The Futures Group International) and other cooperating agencies (CAs). The evaluation will determine what has been accomplished to date for the various interventions, their effect vis-à-vis stated objectives, and their potential for expansion and replication. The primary focus will be on the work being performed by the Futures Group (Youth.now). Other supportive CA contributions are much smaller and of a discrete, time￾limited nature and not designed to be self-sustaining. This will be taken into account in the evaluation process. Evaluation question: How effective has The Futures Group been in the overall management of the ARH activities? The major emphasis will be on the feasibility of the approach developed to improve adolescents access to reproductive health, information, education and services and the development of the models for replication. For example the evaluation team will determine progress to date in terms of the number of young persons accessing ARH services, the effectiveness of the mass communication campaigns and the potential effect the project may have on the community. B. Assess Implementation Strategies Evaluation question: How are service providers, and providers of care being used, to implement the project and have they been effective? The evaluation team will review existing documentation including the project’s MIS system, annual reports, and data information. They will conduct informant interviews with project personnel, stakeholders, service providers, and key MOH and NGO community-based personnel. They will assess progress made at the youth-friendly service delivery points, the effectiveness of the mass communication interventions vis-à-vis changing attitudes and behaviors, the effectiveness of training efforts, and the program’s support for an improved policy environment for adolescent reproductive health. C. Assess the involvement of the Ministry of Health and other sector ministries Evaluation question: How involved have the Ministry of Health and other sectoral ministries been in the operation of the program? Has their contribution been technically adequate? The team will determine the extent that the Ministry of Health and other sectors have contributed and are able to contribute to improving increased access to services, improved knowledge and skills of service providers, and to the implementation of national policies and guidelines in the targeted areas. This will be accomplished through reviewing collected data and reports and conducting interviews with service providers community members, teachers and parents. 57 D. Recommend follow-on activity development With the ARH project reaching its final two years, planning for follow-on activities is crucial. The evaluation team will review the status of the ARH interventions and propose subsequent activity for USAID/Jamaica support of the strategic objective. The review will focus on the interventions that can be shown to have contributed to improved reproductive health of youth and the enabling environment that made them effective. The review will include recommendations for replicating those interventions. This information and any subsequent recommendations will also be used to help USAID/Jamaica formulate its Strategic Plan for FY2005-2009. 5. Team Composition and Participation The evaluation will be conducted over a one-month period by a multi-disciplinary five-person team. The evaluation team should consist of: 1) Team Leader 2) Community Participation Specialist 3) Evaluation Specialist, 4) Youth Consultant, and 5) Local Liaison The Team Leader will be responsible for coordinating the activities of the evaluation team. The team leader will oversee the development of the evaluation schedule and approach, the findings of different team members and coordinate the preparation of the final reports. The team leader will ensure the timely completion of the evaluation report, including recommendations for follow-on activities. This will require approximately three person weeks (six-day workweek) of effort. (November 4-26) The Team Leader will have at least five years of experience in the administration of multi￾faceted adolescent reproductive health projects in developing countries and the Caribbean, and in Jamaica preferably. The team leader will have experience in managing large, multi-disciplinary teams, and the ability to conceptualize and write clearly and concisely. An MPH or equivalent in health, planning, social sciences or similar field is required. The Community Participation Specialist will be responsible for assessing the appropriateness and effectiveness of the community interventions developed through the project. (S)he will examine the congruence between the project inputs developed or supplied through the project at the community level and through the mass media and the Jamaican health system and will assess the functions carried out by the service providers and the potential multiplier effect in light of their training through the ARH project. This individual will also assess the demand for services at the community level based on the intervention/methodology adopted by the project and analyze the extent to which the approach assists in the achievement of increased access and improved knowledge and skills acquired in relation to reproductive health and HIV/AIDS/STIs. The community participation adolescent reproductive health specialist will have at least ten years of experience in developing countries including the Caribbean, and be familiar with the implementation of community based interventions. An MPH or equivalent in health, social 58 science and related areas is needed. This will be approximately three person weeks (six-day workweek) of effort person. (November 4-26, 2002) The Evaluation Specialist will provide expert technical advice and support on evaluation methodology to the evaluation team, including analysis and interpretation of data required to adequately assess project direction and impact to date. The evaluation specialist will also have responsibility for drafting, including tables, graphs and other quantitative data. The evaluation specialist will hold an MPH or equivalent with emphasis on health research/statistics and have extensive experience in evaluating health programs. S(he) will have at least ten years experience working with reproductive and efficient in the use of Power Point, Excel and Word Perfect. This will be approximately three person weeks (six-day workweek) of effort. (November 2-26, 2002) The Youth Consultant will provide Technical Assistance for the evaluation. S(he) will provide advice, information to the team on an ongoing basis which reflects the perspectives of the beneficiary group and the realities of adolescent lifestyle and behaviors in Jamaica. The Local Liaison Manager will provide logistical support for the evaluation. S/he will work with the Chief of Party and team members in preparing for field visits, meetings/consultations in Jamaica, and support the team substantively in the management/administration and documentation of the evaluation. 6. Technical Direction The Director, Office of General Development (and/or her designee), will serve as the Evaluations Cognizant Technical Officer for the evaluation and shall provide formal technical direction and general guidance during the performance of this effort. USAID/J will assist with making arrangements for visits to selected youth-friendly sites, meetings with key partners from the Ministry of Health, other sectoral ministries, and other institutions as appropriate. 7. Report/Deliverables The consultant team will conduct an initial briefing with members from the Office of General Development, to discuss the SOW and Plan of Action for the evaluation and regularly thereafter to brief USAID/J on progress. A draft report with recommendations for the design activity will be submitted to USAID/J one day prior to the report presentation, to be held on November 22. The report will be finalized and five copies (along with a diskette in Word 97) will be submitted November 25. It should be no more than 30 pages long. In accordance with requirements of the statement of work, the evaluation contractor will provide the following deliverables: 1. An evaluation report which will include: a. findings on status of project implementation and effectiveness to date; b. assessment of potential expansion and replicability of project interventions c. recommendations (if appropriate) for midcourse changes in the present project and issues/interventions concerning future direction ( follow-on activity) 59 2. The evaluation report should include the following sections: · Executive summary · Introduction · Background of the problem · USAID's assistance approach: The ARH Project · Findings · Conclusions/lessons learned · Recommendations for Future directions · Annexes 60 ANNEX B LIST OF PERSONS CONTACTED Mrs. Deloris Brissett Acting Chief Education Officer, Ministry of Education, Caenwood Complex 37 Arnold Road, Kingston 5 Mrs. Burrell CHANGE Project/AED Rural Family Support Organization May Pen, Clarendon Ms. Natalie Campbell Director, National Center for Youth Development Ministry of Education, Caenwood Complex 37 Arnold Road, Kingston 5 Dr. Marian DeBruin Acting Director, CARIMAC, University of the West Indies Mrs. Lois Hue Director, Youth & HIV Programs, Red Cross Society of Jamaica Central Village, St. Catherine Mr. Don Levy Project Director, Youth.now Mrs. Jan Lopez Program Director, Addiction Alert 57 East St. Kingston Mrs. Kathy McClure Consultant Mr. Gregory McClure President, Ms. Joan Robb Director, Client Services, Dunlop Corbin Communications Ltd., 6 Oxford Road, Kingston 5 Mrs. Cecile Minott Jamaica Foundation for Children Old Hope Road, Kingston 6 61 Dr. Olivia McDonald Director, Mrs. Janet Davis IEC Officer, National Family Planning Board 5 Sylvan Avenue, Kingston 5 Mrs. Sarah Newland-Martin General Secretary – YMCA Mrs. Shirley Oliver-Miller Margaret Sanger Center International Mr. Errol Alexis Master Trainers Refresher Training Course II FDR Hotel – Runaway Bay Mr. Aaron Panther Moravian Church in Jamaica 3 Hector Street, Kingston 5 Nurse Pariola FAMPLAN Ms. Gail Hoad 57 East St. Kingston Dr. Richard Reid 21 Hope Road, Kingston 10 Mrs. Deborah Patrick Planning Institute of Jamaica Mr. Walter James 10 – 16 Grenada Way, Kingston 5 Mrs. Claudette Pious Executive Director, Children First Mr. Damian Brown 9 Monk St. Spanish Town Mrs. Peggy Scott Executive Director, FAMPLAN 14 King Street, St. Ann’s Bay Mr. Orville Ranglin Guidance Counsellor, Nain All Age & Junior High School Nain, St. Elizabeth Mr. Joseph Robinson Executive Director, Ms. Karlene Temple Administrator, ASHE 143 Mountain View Avenue, Kingston 3 62 Mrs. Beryl Weir Executive Director, Ms. Sheryl Morris Director, Field Operations Mrs. Charmaine Johnson Women’s Center of Jamaica Foundation 42 Trafalgar Road, Kingston 10 Ministry of Health Personnel Head Office Dr. Barry Wint Chief Medical Officer Dr. Peter Figueroa Director, National HIV /STI Program Dr. Deanna Ashley Director, Health Promotion & Protection Division Dr. Karen Lewis-Bell Director, Family Health Services Ms. Ingrid Thame Quality Assurance Advisor, QAP Ms. Thelma Campbell Chief Nursing Officer, Ministry of Health Mrs. Marcia Hyman-McKay Chairperson, National Curriculum Committee Nursing & Midwifery, Ministry of Health Regional Health Authorities Dr. Jacqueline Gernay Southeast Regional Technical Director Dr. Michael Coombs Southern Regional Technical Director (SRHA) Parish Health Departments and Health Centers Dr. Donna Royer-Powe Kingston and St. Andrew Health Department Dr. Cotterell-Grant May Pen Health Centre Mrs. M. Brown Nutritionist, Clarendon Health Department 63 Ms. Serena Powell Staff Nurse, Clarendon Health Department Ms. Carlisa Pearson Health Educator, Clarendon Health Department Ms. Maureen Watson Milk River Health Centre Ms. Laurett Wright Assistant Dental Nurse, Clarendon Health Department, Mr. Canute Thompson Clarendon Health Department Group of 8 Peer Educators St. Elizabeth Mrs. Francis Public Health Nurse, Junction Health Centre Junction, St. Elizabeth Miss Faith Salmon Public Health Nurse, Balaclava Health Centre Balaclava Health Center Mrs. Dawkins Ellis Dental Nurse Ms. Rosina Powell Community Health Aide Mr. Ivor Patrickson Public Health Inspector Ms. Jennifer Chambers Staff Nurse Mrs. M. burnett-Watson Dental Assistant Ms. Oneika Cunningham Community Peer Education Group of 7 Peer Educators Youth.now Personnel Dr. Pauline Russell-Brown Chief of Party Ms. Catherine Lane Program Manager Miss Karen Thaxter Training Coordinator Mr. Newton Wynter Advocacy Coordinator Mrs. Viviene Kerr-Williams Parish Officer (Clarendon) Mr. Wilbert Rowe Parish Officer (St. Elizabeth) Miss Yulonda Smith Adolescent Advisor Mr. Hylton Grace Adolescent Advisor Mr. Peter Jackson Finance & Administration Manager Ms. Rosemary Nethersole Programme Assistant / Administration United States Agency for International Development Ms. Mosina Jordan Mission Director Mrs. Margaret Sancho Director, Office of General Development Mrs. Jennifer Knight-Johnson Project Management Specialist 64 ANNEX C SO 3 RESULTS FRAMEWORK 65 ANNEX D Youth.now NGO Subgrants Project Profile Women’s Center of Jamaica Foundation “Adolescent RH Seminars for Young Men at Risk” JA $2.7 million 7/1/01—5/31/03 (extended) Purpose/Activities --Decrease early sexual initiation and parenthood by expanding RH knowledge of young men at risk --Conduct a series of seminars on adolescent RH in Kingston and Montego Bay areas --Train 50 leaders from youth clubs/organizations in these areas to address RH issues with peers --Impact evaluation to assess what young men pass on and extent to which they start accessing RH services Achievements to Date --Initial seminars conducted in both areas --Baseline conducted among young men in project areas in Montego Bay (Baseline abandoned in Kingston owing to pre￾election violence) --26 young men in MB and 20 in Kingston trained in summer ‘02 --Initial outreach by trained youth underway Comments --Limited follow-up planned for trainees or their peer contacts—emphasis of subgrant is on training, not outreach activities --No specific targets for outreach activities --Project will end about nine months after initial training completed; current time-frame requires that end-line survey be conducted relatively soon after outreach begins. ___________________________________________________________________________________________________________ FAMPLAN “Urban Youth First” JA $6.7 million 7/1/01—6/30/04 --Improve RH of adol. by increasing access to/use of RH services in downtown Kingston --Use two youth promoters to provide outreach, recruit clients --Increase number of youth using RH services at upgraded clinic --Support RH education in area schools to improve knowledge and skills --Provide “wide range” of RH/FP services and improve quality of services at renovated clinic --In Yr. 2, train peer educators to provide OCs/condoms --Baseline conducted among 350 youth in 7 schools --RH/LS education provided to over 1000 students in 9 schools on a regular basis --Training conducted for staff --2002 Q2: 82 adol. FP visits; 16 counseling; 73 antenatal; 8 postnatal 67 --2002 Q3: 41 adol. FP visits; 3 for counseling; 86 antenatal and 9 postnatal (record-keeping for FP in this period was incomplete.) --No quantitative objectives in original proposal; overly elaborate M&E plan prepared after lengthy negotiation. Targets expressed in % change from baseline and not in absolute numbers. -- Clinic renovation stalled for security reasons; poor condition of clinic and lack of separate space for adol. is thwarting original project objective with respect to services. Nonetheless, a small number of youth are accessing the services. ____________________________________________________________________________________________________________ _____________________ Project Profile Children First “Change for Life” JA $9.6 million 4/1/02-9/30/04 Purpose/Activities --Project complements EU￾funded “Youth Wellness Centre” which will provide RH services and knowledge and skills for out-of-school youth. --Deliver RH education to 1000 out-of-school adolescents --Develop ARH manual for at risk out of school youth --Support information technology and small business training --Develop advocacy and public education campaign to promote RH behavior change --Develop youth magazine --Support/train 500 caregivers and service providers --Develop youth-friendly health facilities locally --Provide counseling and appropriate contraceptive use --Assess impact on risk behavior; indicators include 80% of target pop receiving life skills education; increased demand for condoms; use of services by youth, reduction in risk behavior, 80% service providers and care-givers trained, inter alia. 68 Achievements to Date --Approx. 50 older youth enrolled in IT/business training program; some assisted with summer jobs --Approx. 200 UAP and other youth on site receive RH education/life skills. --Youth from community come to CF for health care, inc. STIs. About 10 youth per week are referred (not clear how many for RH services.) Comments --Proposal is highly diffuse with long lists of objectives, indicators, activities --Division of labor for EU and Youth.now grants is confusing; possible overlap/double funding? --Some YN-funded components are less closely linked to RH --As yet, no RH clinical services have as yet been established under EU grant as planned; currently, counseling and referrals are all that are provided. There are no condoms available on site. Future plans include part-time medical staff. --Local health facilities have not agreed to release staff from potential referral facilities for training. Plan is to train dental staff instead. 69 ____________________________________________________________________________________________________________ ____________________ Project Profile Jamaica Foundation for Children “Upgrading Adolescent Hotline Services” JA $2.9 million 7/1/01—9/30/02 (extended) Purpose/Activities --Increase capacity to deliver telephone counseling, information and referral --Expand hotline to include 24 hour fact-line. --Provide a minimum of 30 messages on RH information (sic) --Train 20 new telephone counselors and upgrade 20 existing volunteer counselors --Referral to relevant services --Redesign and maintain database to add RH component Achievements to Date --Factline evaluated; present messages revised and new messages developed --In June 2002, 15 new and 10 old volunteers trained in sexuality, HIV, relationships, counseling skills --During the project period, the hotline fielded a total of 2325 calls, of which 84% were female; 72% were youth. Comments --This is a focused grant providing a useful and high quality service. --Strengthening the RH counseling provided by the hotline service appears to be good value. --Number/proportion of calls relating to RH issues is not entirely clear. ____________________________________________________________________________________________________________ _____________________ YMCA “Strengthening Partnerships in support of ARH” JA $6.4 million 2/1/02—9/30/04 --Improve RH knowledge and behaviors for youth served by the Y (about 500.) --Establish a youth-friendly service site for all Y users with RH education and counseling; on-site condom services, and referral to health facilities --Provide boys in UAP program with similar RH education and services; reach their parents with a parenting education program. --Improving referral between the Y and selected clinical sites is an important program element. --Research involving complex questions and surveys to evaluate these interventions. 70 --Project recruited staff are providing RH/LS education to 182 UAP students. --X boys referred for various, unspecified services to Bethel Baptist; 2 referred for HIV/VCT. --Monthly parent-teacher meetings. --Baseline planned for January 03. --Status of services for broader Y population unclear. --Although condoms are available on site, none have been distributed. As a Christian organization, Y prefers not to openly distribute condoms; boys must request them. --Most referrals to date appear to have been for general medical reasons. ____________________________________________________________________________________________________________ _____________________ Project Profile Jamaica AIDS Support “Life fi Yute” JA $1.1 million 7/15/02-7/15/03 Purpose/Activities --Decrease transmission of HIV/STIs and improve RH among marginalized urban youth --Approx. 40 marginalized youth from three sports clubs in Montego Bay and 120 from the Y in Kingston, including some who are sexual minorities and in transactional relationships, will be targeted for sports and performing arts activities to improve their “life coping skills” and transmit SRH information. --Evaluation based on increased knowledge, VCT requests and requests for condoms from JAS HQ. Achievements to Date N/A Comments --Proposal went through five revisions and resubmissions owing to weak design. --Version of proposal reviewed included extensive musical and sports equipment --Project is working with youth from Kingston YMCA Amy Bailey Center. YMCA subgrant is also working with 32 youth from Amy Bailey—possible overlap in target audience? 71 ____________________________________________________________________________________________________________ _____________________ Children for Community Change “3C’s Life Skills Public Education Project” JA $1.9 million 8/1/02-8/31/03 --Reduce HIV transmission among youth in Kingston/St. Andrew --Provide HIV/life skills education after school to 100 youth from Mandela Terrace community --Train 20 peer educators --Strengthen capacity of local performing arts group to do public/community education; conduct 30 shows in 7 communities increasing awareness in 2000 youth and community members --Improve capacity of 3C’s to deliver RH programs N/A None 72 ____________________________________________________________________________________________________________ _____________________ Project Profile Nurse Louise Pitter Clinic JA $5.4 million 9/1/02-9/30/04 Purpose/Activities --Address need for HIV education and counseling, FP, among youth in community --Provide RH education for three hours per week to 120 teens 10-19 in community and St. Annie’s High School every year. --Upgrade current facilities for testing and treatment of STI’s/HIV, in collaboration with the Food for the Poor Clinic --Train 20 peer counselors. Achievements to Date N/A Comments ----- ANNEX E Summary Training Chart