Youth-policy.com One-year Evaluation Report, October 2005 1 youth-policy.com Better Policy, Healthier Youth One-Year Evaluation Report October, 2005 This publication was produced for review by the United States Agency for International Development. It was prepared by the POLICY Project and YouthNet. Youth-policy.com One-year Evaluation Report, October 2005 2 Acknowledgments This evaluation was carried out by a team that includes, from the POLICY Project, Nancy Murray, Meghan Corroon, James E. Rosen, and Doug Willier; and, from YouthNet, Sharifah Tahir and Haguerenesh Woldeyohannes. The authors wish to thank those youth-policy.com users who participated in the evaluation. The POLICY Project works with developing country governments and civil society organizations to promote a more supportive policy environment for family planning/reproductive health, HIV/AIDS, and maternal health programs and services. The POLICY Project is funded by the U.S. Agency for International Development under Contract No. HRN-C-00-00-00006-00. POLICY is implemented by Futures Group in collaboration with the Centre for Development and Population Activities (CEDPA) and Research Triangle Institute (RTI). www.policyproject.com YouthNet is a global program funded by the U.S. Agency for International Development to improve reproductive health and HIV prevention needs of youth ages 10 to 24. YouthNet conducts research; works to improve and strengthen youth programs, services, and policies; and disseminates and promotes information, tools, and evidence-based resources. www.fhi.org/youthnet The U.S Agency for International Development (USAID) is a federal agency that extends assistance to countries overcoming global challenges, recovering from disasters, trying to escape poverty, and engaging in democratic reforms. USAID supports both the POLICY Project and YouthNet. www.usaid.gov Youth-policy.com One-year Evaluation Report, October 2005 3 Table of Contents Acknowledgments Summary I. Introduction A. Background B. Objectives of the Evaluation II. Evaluation Methodology A. Quantitative B. Qualitative C. Cost Analysis D. Limitations III. Findings A. Site Usage B. Impact C. Cost and Cost-Effectiveness IV. Discussion and Future Directions Appendices Appendix A: Site Diagram Appendix B: Selected Examples of the Impact of Youth-policy.com on YRH Policy Youth-policy.com One-year Evaluation Report, October 2005 4 Summary Launched in June 2004, the youth-policy.com website uniquely meets the needs of developing country policymakers for accurate, comprehensive, and timely information on youth reproductive health (YRH) policy. The U.S. Agency for International Development (USAID) funds the website as a joint effort of the POLICY Project and YouthNet. An evaluation of the site’s first full year of operation found: • The site has reached over 17,000 users. • By the end of the first year, site use has risen steeply to over 2000 users per month, more than triple the volume of the site’s early months. • Users come from a wide range of developing countries. Of the 50 most active countries, 29 are developing nations. • A healthy balance exists among the most popular features of the site. The database search is the most popular feature, and users show considerable interest in the supporting technical material, especially the fact sheets and the tools and links. • Based on limited anecdotal information, it appears that key audiences are using the site for policy impact. • The prospective cost of maintaining and making incremental improvements to the site is $1.41 per unique user, at current levels of use. To continue to make the site an attractive destination for the international community of practitioners working on and for YRH policy, the site team plans to: • intensify efforts to build site utilization by the field, with a focus on developing country users; • better document the field experience to improve our knowledge of impact; • update and expand current content, especially the policy database; • explore opportunities for increasing the interactive features of the site; • continually improve the site functionality so that it remains a useful and attractive resource for the field; • continue efforts to link the site with other important sites working on youth, reproductive health, and policy; and • reach out to other groups working on these issues. Youth-policy.com One-year Evaluation Report, October 2005 5 I. Introduction Youth-policy.com recently passed its one-year anniversary as an online resource on youth reproductive health (YRH) policy. This evaluation attempts to gauge the success of the site in achieving its objectives. It combines a quantitative analysis based on web statistics with qualitative, anecdotal information on who is using the site and for what reasons. The youth￾policy.com team from the POLICY Project and YouthNet carried out the evaluation. Evaluation results will help to measure the site’s impact on YRH policy and feed into site upgrades and improvements. A. Background Launched in June 2004, youth-policy.com is a joint effort of the POLICY Project and YouthNet. The U.S. Agency for International Development supports both projects. The overall goal of youth-policy.com is to promote the spread of good YRH policies—and, by extension, improve YRH outcomes—around the world. Hence, the website’s motto: Better Policy, Healthier Youth. The website grew out of the desire to find cost-effective ways to extend the reach of USAID’s support for policy development. Experience had shown that country-level policymakers were often hindered by lack of easy access to examples of policies from other countries and by the absence of a comprehensive framework for thinking about YRH policy. A needs assessment commissioned by the POLICY Project’s Adolescent Working Group in early 2003 confirmed the potential usefulness of a website that would address these gaps. A formal collaboration with the YouthNet Project was established in late 2003, and the website team began designing the site and developing the content. To ensure participation from a broad range of groups and to maintain high technical quality of site content, the team carried out a series of consultation meetings with experts in the various key elements of YRH policy, including pregnancy prevention, HIV/AIDS, and so forth. Subsequently, in March 2004, the team organized a one-day consultative meeting and got feedback on an early version of the site from over 30 participants from various organizations, including USAID and its cooperating agencies, other international donor and technical assistance groups, and developing country youth policymaking agencies. The site was officially launched in early June 2004 at the annual meeting of the Global Health Council, whose theme that year was “Youth and Health.” The specific objectives of the website are to • help developing countries design comprehensive YRH policies; • provide guidance in using good practice language in the development of new or updated YRH policies; • increase awareness of the current state of YRH policies and how to improve them; and • promote the exchange of information on YRH policies. Youth-policy.com One-year Evaluation Report, October 2005 6 The core of youth-policy.com consists of a searchable database of full-text policies addressing YRH, a guide that describes the key elements of YRH policy and guiding principles for good policy language; and links to policymaking tools, case studies, papers, reports, and articles (see Appendix A). The website aims to increase the reach and utilization among target audiences, namely advocates for YRH; young people; policymakers in the government, NGO, or private commercial sector; providers of technical assistance in the design of YRH policies; representatives of donor or international agencies; program managers of a YRH program; and providers of YRH services. Since the launch of the website in June 2004, several new features have been added, including 40 new policies in the database, an upgraded search function, updates to fact sheets, dozens of new and updated links and tools. The team prepared and disseminated three quarterly newsletters and three quarterly evaluation reports, and carried out a variety of marketing activities. B. Objectives of the Evaluation This evaluation is an attempt to gauge the success of the site in achieving its objectives. The team plans to use evaluation results to help measure the site’s impact on YRH policy and feed into site upgrades and improvements. The main evaluation questions, grouped into process and impact evaluation questions, are: Process questions • Are people using the website? • How are they using it? • Are we reaching the intended audiences? • What was the cost of running the site? Impact questions • Has the website raised awareness of the current state of YRH policies and how to improve them? • Has the website served to encourage the exchange of information on YRH policies? • Has the website helped countries to develop more and better YRH policies? II. Evaluation Methodology The youth-policy.com team from the POLICY Project and YouthNet carried out the evaluation using a combination of quantitative analysis and qualitative, anecdotal information. A cost analysis provides some information on the cost of running the site. Youth-policy.com One-year Evaluation Report, October 2005 7 A. Quantitative Analysis The quantitative analysis is based on standard site user statistics covering the period July 1, 2004 – June 30, 2005. These statistics allow an estimate of trends in the number of site users, their country of origin, the most popular referring sites, and the most popular web pages. We chose July 1, 2004 as the start date for analysis because July was the first full month of operation of the site after its launch in early June 2004. B. Qualitative Analysis Over the year that the site has been in operation, the team also collected information on site use directly from selected users. This information is derived primarily from personal communication between site users and team members, and forms the basis for answering questions about site impact. C. Cost Analysis The team also carried out a prospective cost analysis based on the estimated time and resources needed to continue operating the site at an adequate level. D. Limitations of the Evaluation The evaluation has some important limitations. Information from personal communication with site users is not representative of the broader set of site users and must be interpreted with caution. Measuring the impact of any website is difficult and it may still be too early to adequately measure the impact of youth-policy.com. Moreover, the difficulties in measuring activities related to policy development and implementation are well-known. Despite such difficulties, the team felt it was worthwhile to attempt to go beyond the standard web statistics to gain a better sense of whether the site is making a difference. III. Findings This section summarizes findings on site usage, impact, and cost. A. Site Usage These summary web statistics are from the one-year period from July 1, 2004 to June 30, 2005. User sessions. The number of user sessions gives a sense of the volume of users of the site. User sessions are unique, and thus the team did not double count users who stay on the site and use multiple pages. For the year, the site had 17,190 user sessions. User sessions remained stable at about 700 per month during the first six months of the site’s operations, and then steadily began to climb beginning in December 2004. Since then site use has about tripled, to over 2,000 users per month (see Figure 1) or about 73 users per day. Youth-policy.com One-year Evaluation Report, October 2005 8 Figure 1: Youth-policy.com User Sessions, July 2004-June 2005 636 649 684 694 667 963 1349 1744 1960 2360 2300 2248 0 500 1000 1500 2000 2500 July Aug Sept Oct Nov Dec Jan Feb Mar Apr May June Month No. of Sessions Most popular pages. This statistic identifies the most popular website pages and how often users accessed them. As shown in Table 1, the most popular features are the database search—with nearly 5,000 views—followed by the home page, fact sheets, tools and links, newsletters, the key elements guide, and the making policy section. Users also showed interest in finding out more about youth-policy.com and in getting involved, including joining the youth-policy.com network. Table 1: Most Popular Pages, Youth-policy.com, July 2004 – June 2005 Page No. of Views Database Search 4,883 Home Page 4,269 Fact Sheets 2,693 Tools and Links 2,188 Newsletters 943 Key Elements Guide 830 Making Policy 614 Contact Us 585 About 469 Get Involved 392 Join Network 241 Feedback 55 Youth-policy.com One-year Evaluation Report, October 2005 9 Table 3: Other Developing Countries Among the 50 Most Active Countries, Youth-policy.com, July 2004–June 2005, by Level of Activity from Highest to Lowest Cambodia, India, Pakistan, Nepal, Trinidad and Tobago, Viet Nam, Yugoslavia, Mexico, Kenya, Rwanda, Peru, Malaysia, Tanzania, Myanmar, Morocco, Romania, Russian Federation, Jamaica, Croatia, Guyana, Estonia Most active countries. This statistic identifies the top locations of users by country. The country of the user is determined by the suffix of their domain name. This information is based on where the domain name of the visitor is registered, and may not always be an accurate identifier of the actual geographic location of this visitor. According to these statistics, about 73 percent of users are from the United States. Other countries represent 9 percent of users and another 18 percent of users are not identified by country. As Table 2 shows, of the top 15 most active countries, eight are in the developing world. Another 21 developing countries are among the top 50 most active countries (see Table 3). Over the year, the number of developing countries on the top 50 list has risen from 26 to 31. Organizational usage. The statistics on organizational usage give a very rough sense of the organizational affiliation of site users. For example, those users with a .org in their web address are classified as nonprofit; those with .gov as government, etc. Most site users, 80 percent, have either a .com or .net address. Smaller percentages have a .edu (10%), .org (9%), or .gov (0.6%) suffix. These proportions have essentially remained stable over the year of the site’s operation. Top referring sites. As with any website, users can access the site directly or via a link at another website. The statistics on top referring sites give a sense of where users are coming from when they enter the site. Approximately 70 percent of youth￾policy.com users visit the website directly and do not come through a referring site. Of the remaining one-third of users who do enter via a link, the top seven referring sites are Google, Yahoo, the HIVinsite of the University of California at San Francisco, FHI, MSN, UNESCO Bangkok, and Whois Source (a search engine) (see Table 4). Although search engines such as Google accounted for only about 25 percent of referrals during the first quarter of the website’s operation, by the fourth quarter of operations search engines accounted for about 90 percent. Table 2: Youth-policy.com Most Active Countries: July 2004 – June 2005 1. United States 2. United Kingdom 3. Australia 4. Canada 5. Germany 6. Netherlands 7. Seychelles 8. Botswana 9. Indonesia 10. Zimbabwe 11. Thailand 12. South Africa 13. Zambia 14. Sweden 15. Philippines Table 4: Youth-Policy.com Top Referring Sites: July 2004 – June 2005 1. Google 2. Yahoo 3. HIVinsite (U.C. San Francisco) 4. Family Health International 5. MSN 6. UNESCO Bangkok 7. Whois Source (search engine) Youth-policy.com One-year Evaluation Report, October 2005 10 B. Impact The site’s ultimate goal is to have an effect on YRH policies in developing countries. The following paragraphs describe how the site may have possibly had an impact according to our objectives as laid out in the introduction above (see Appendix B for a summary of these examples). Africa. Members of the Africa Regional Youth Initiative’s (ARYI) Youth Policy Group used the website to research and examine YRH policies in countries with member networks, including in Tanzania, Uganda, South Africa, Ghana, Zambia, Nigeria, and Kenya. ARYI used the information to create awareness among African youth on reproductive health rights, policy, and programs and to monitor the implementation of YRH policy. Botswana. The Director of the Health Research Unit of the Ministry of Health drew on the experience on youth-policy.com in developing a database of health policies in Botswana. Haiti. Cooperating agency technical assistance staff and local NGOs searched the website database on youth VCT and used the fact sheet on VCT as background for policy work in Haiti. Information from the website fed into the development of a policy note on VCT and youth and ultimately enhancement of national VCT policy as it addresses youth. India. Cooperating agency technical assistance staff searched the site for information on youth programs and found a “great link” to information about youth programs in India. Namibia. Cooperating agency technical assistance staff used the policy database to examine provisions of the National Policy for Reproductive Health as they relate to RH information for adolescents. This information was used to support the implementation of an action plan on adolescent reproductive health prepared by a team of Namibian policy and program experts. Zambia. World Bank staff used youth-policy.com to examine policies in Zambia related to youth and HIV in preparation of a case study as part of an assessment of the Bank’s work on youth and HIV in Africa. The ultimate goal is to improve the program and policy environment for youth and HIV and to increase funding and effectiveness of programs. C. Cost and Cost-Effectiveness The cost of maintaining and updating the site is important to consider in relation to its overall impact and sustainability. We calculated the prospective cost of maintaining and making incremental improvements to the site at a level that would maintain its freshness and utility. The total yearly direct cost comes to $39,000. Under the conservative assumption that site use will remain constant at about 2,300 unique users per month, the cost per unique user for an entire year would be $1.41. Staff and consultant time accounts for almost the entire cost of running the site, and includes technical maintenance, administration, management, and coordination; adding new policies to and maintaining the database; updating and making selected additions to the “Making Policy” and “Tools and Links” sections of the site; communications, including preparing and disseminating a quarterly newsletter; and evaluation. Youth-policy.com One-year Evaluation Report, October 2005 11 IV. Discussion and Future Directions The main conclusions from the findings are: • The site reached over 17,000 users in its first year of existence. • Encouragingly, site use is rising, a trend that began about six months after the site launch. • Another positive sign is that a wide range of developing countries is using the site, and that this number has steadily increased over time. • At the same time, most of the users apparently are from the United States and other developed countries. Although our site audiences include technical assistance or funding agencies in developed countries, we would hope to see larger numbers of users from the developing world. • A healthy balance exists among the most popular features of the site. The database search is the most popular feature, an indication of the unique value of this component of the site. There is also considerable interest in the supporting technical material, especially the fact sheets and the tools and links. The newsletter, mainly a marketing tool, has been surprisingly popular, with almost 1,000 views for just two editions. • Because of the limitations of the data, few conclusions can be drawn about site impact. However, it does appear that key audiences are using the site for policy impact. • Under conservative assumptions, the prospective cost of maintaining and making incremental improvements to the site is $1.41 per unique user, at current levels of use. Our aim now is to consolidate the experience of the past year and continue to make the site an attractive destination for the international community of practitioners working on and for YRH policy. The numbers and trends clearly point to this site as a useful destination for many. Our goal is to build on this awareness and to mainstream the site in the consciousness of all those out there working on these important issues. Important elements of this effort will be to: • intensify efforts to market the site to the field, with a focus on developing country users; • better document the field experience to improve our knowledge of impact; • update and expand current content, especially the policy database; • explore opportunities for increasing the interactive features of the site; • continually improve the site functionality so that it remains a useful and attractive resource for the field; • continue efforts to link the site with other important sites working on youth, reproductive health, and policy; and • reach out to other groups working on these issues Youth-policy.com One-year Evaluation Report, October 2005 12 APPENDIX A: Site Diagram Youth-policy.com SITE INDEX HOMEPAGE NEWSLETTER POLICY DATABASE MAKING POLICY TOOLS & LINKS GET INVOLVED ♦ Policy Database Contains 115 policies from 41 countries. Searchable by key RH elements such as STI/HIV infection prevention and reduction of harmful traditional practices; policy type, such as HIV/AIDS policies and youth RH policies; country; language; or any combination of these fields. ♦ Key Elements and Good Practice Guide Presents a framework for good YRH policy. Describes the key elements a good YRH policy should contain and provides guiding principles for good policy language. ♦ Fact Sheets Provides detailed information on a range of issues related to the key elements identified as essential for responding to YRH needs. Topics include: VCT, Female Genital Cuttin, Contraception, ECP, Maternity Care, Nutrition, STIs. ♦ Policymaking Tools for YRH policymaking, case studies of policymaking, and related papers, reports, and articles. ♦ Links Provides links to useful YRH policy sites worldwide and to related policy and advocacy documents, case studies, and tools, including:. • Best of the web • Organizations working on YRH policy • Tools for YRH policymaking • Case studies • Key papers, reports, and articles ♦ YRH Network Directory Provides country contacts and the option to post yourself as a contact person. ♦ Send Us Your Policy To be added to the compendium, and featured on the site. ♦ Newsletter Signup To receive updates on policy developments, upcoming workshops, etc. ♦ Announcements, Events, News KEY ABOUT DEFINITIONS US/CONTACT INTENDED SITE USERS Policy Database Search Engine Youth-policy.com One-year Evaluation Report, October 2005 13 Appendix B: Selected Examples of the Impact of Youth-policy.com on YRH Policy Country or Region Users Policy Impact Ultimate impact Africa Young Africans advocating for YRH Awareness-raising on RH rights, policies, and programs; implementation of YRH policy Improve the well-being of African youth and their families Botswana Ministry of Health research official Development of health policy database Improve the health of all people in Botswana, including youth Haiti Cooperating agency technical assistance staff, local NGOs Development of guidelines on Youth and VCT; enhancement of national VCT policy as it addresses youth Expand access to VCT India Cooperating agency technical assistance staff Improved knowledge of YRH programs and policies in India Expand access to YRH information and services Namibia Cooperating agency technical assistance staff, local NGOs Implementation of action plan on ARH policy Expand access to ARH information and services Zambia World Bank staff Implement in the Bank’s work on HIV and youth Mobilize greater funding for HIV and youth work and improved policy environment