This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact under the Liberia Strategic Analysis activity. Its authors are Sarah Auten, Amit Dabla, and Mark Bassie. Advancing Youth Project Performance Evaluation Final Report ADVANCING YOUTH PROJECT PERFORMANCE EVALUATION: FINAL REPORT A USAID/LIBERIA PROJECT TO PROVIDE ALTERNATIVE BASIC EDUCATION FOR LIBERIAN YOUTH AGES 13 - 35 MAY 2016 Liberia Strategic Analysis Award No: AID-669-C-16-00002 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms........................................................................................................................................................................1 Executive Summary.......................................................................................................................................................3 Evaluation Purpose & Evaluation Questions...........................................................................................................9 Project Background ....................................................................................................................................................11 Evaluation Methods & Limitations...........................................................................................................................16 Findings, Conclusions & Recommendations.........................................................................................................19 Findings......................................................................................................................................................................19 Conclusions..............................................................................................................................................................48 Recommendations..................................................................................................................................................53 Annexes.........................................................................................................................................................................61 Annex I: Evaluation Statement of Work...........................................................................................................62 Annex II: Evaluation Methods and Limitations.................................................................................................68 Annex III: Data Collection Instruments............................................................................................................73 Annex IV: Sources of Information......................................................................................................................90 Annex VI: Disclosure of any Conflicts of Interest .........................................................................................93 Annex VI: AYP Project Overview ......................................................................................................................96 Annex VII: OLA and EGMA Subtest Results................................................................................................. 101 Annex VIII: OLA/EGMA Baseline Study 2015/16......................................................................................... 103 1 ACRONYMS ABE Alternative Basic Education AE Alternative Education AYP Advancing Youth Project CBO Community-based Organization CEO County Education Officer CESLY Core Education Skills for Liberian Youth CLA Collaboration, Learning and Adapting COP Chief of Party CSI Centers for Skills Innovation DEO District Education Officer DO Development Objective EDC Education Development Center, Inc. EGMA Early Grade Mathematics Assessment EQUIP Education Quality Improvement Program EU European Union FGD Focus Group Discussion GBCC Grand Bassa Community College GOL Government of Liberia GYB Grow Your Business IAI Interactive Audio Instruction ICA Institutional Capacity Assessment ICD Institutional Capacity Development KII Key Informant Interview LIYONET Liberian Youth Network LMA Labor Market Assessment LOIC Liberia Opportunities Industrialization Center LRC Learning Resource Center LSA Liberia Strategic Analysis M&E Monitoring and Evaluation MOE Ministry of Education MOYS Ministry of Youth and Sports NAEAL National Adult Education Association of Liberia 2 NGO Non-governmental Organization OCA Organizational Capacity Assessments OJT On-the-job Training OLA Out-of-school Literacy Assessment PIDS Performance Indicator Database System PMP Performance Monitoring Plan PPD Program and Project Development PQS Program Quality Standards PTA Parent-Teacher Association PTI Peer Training Innovators RTTI Rural Teacher Training Institute SI Social Impact TVET Technical and Vocational Education and Training TWG Technical Working Group UNIDO United Nations Industrial Development Organization USAID United States Agency for International Development VSLA Village Savings and Loan Association VTC Vocational Training Center YDLC Youth Development and Livelihoods Coordinator YMCA Young Men’s Christian Association 3 EXECUTIVE SUMMARY Social Impact (SI) is an international development consulting firm whose mission is to create dramatic improvements in the performance of organizations and programs working to enhance the social and economic well-being of people around the world. SI is contracted by USAID/Liberia to implement the Liberia Strategic Analysis (LSA) Activity, a 5-year contract to provide support to USAID/Liberia. Through LSA, Social Impact conducted a performance evaluation of the Advancing Youth Project (AYP), a five-year, $39.8 million activity implemented by Education Development Center Inc. (EDC) in six Liberian counties. AYP works with the Liberian Ministry of Education (MOE) to increase access to quality alternative basic education (ABE) classes, social and leadership development opportunities, and livelihood skills training for out-of-school youth ages 15 – 35, with limited literacy and numeracy skills. This performance evaluation examines the efficacy and sustainability of AYP and documents lessons learned in order to inform the design of future education projects of a similar nature. The following questions are the key points on which the evaluation focused, in instrument design, analysis, and developing findings, conclusions, and recommendations. Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? o To what extent were the Government’s expectations and policy needs met with AYP? o How likely will the Government be to use the ABE materials in the future (confirm that the Ministry has provided accreditation and certification for the alternate learning program and ABE)? o How are communities proposing to sustain the efforts financially or in kind? Youth Targeting  What strategies worked best for recruiting and retaining out of school youth?  What interventions were most successful in helping to create positive outcomes (i.e., completion of course-level 1-3, internship/employment, other livelihood interventions)? Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training? o To what extent did livelihood trainings lead to quality employment for youth?  To what extent did learners improve their literary, numeracy, and work readiness skills to date? This will be informed by EDC's own Out-of-School Literacy Assessment (OLA) and the Early Grade Mathematics Assessment (EGMA), but an independent review of these materials will help inform the Mission and the GOL to become more effective at improving learning outcomes for young adults in the classroom. Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions (based on what has been learned from the accomplishments and challenges of AYP up until now) that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? 4 PROJECT BACKGROUND Many youth in post-conflict Liberia lack basic literacy and numeracy skills, which in turn impacts their employment and livelihood development potential, challenging the country’s economic and social security. In response, USAID developed AYP to pilot an ABE curriculum that offers youth the knowledge and skills they lack. AYP’s Results Framework has two primary strategic objectives: 1) Capacity and Sustainability. MOE and CBOs have the capacity to support and manage alternative basic education programs in Liberia with efficiency, effectiveness, and equality; 2) Access and Quality. Youth have access to clearly defined and relevant education and livelihoods pathways. EVALUATION METHODOLOGY Evaluation Schedule  Week 1: Background review of key documents related to AYP  Weeks 2 – 5: Field work gathering qualitative data from key respondents, including 12 learning sites in four counties, to gain a deeper understanding of the implementing environment  Weeks 6 – 7: Data analysis and report drafting, including presentation of initial findings  Weeks 8 – 10: Report review and finalization, followed up 3 months later by utility review Field work included 61 semi-structured Key Informant Interviews (KIIs) and 14 Focus Group Discussions (FGDs) with 185 total respondents. Data collected during KIIs and FGDs were captured and processed systematically using to a coding system for each topic area. These data were analyzed by the team using an inductive approach; this process informed development of key findings, conclusions and recommendations that make up the bulk of this report. DATA LIMITATIONS Limited Access to Data – The original performance evaluation design planned to combine qualitative data with analysis of quantitative data collected by AYP during implementation. However, the evaluation team had limited access to specific elements of the quantitative data, including the raw data on institutional and organizational capacity building and on learner outcomes. While this did limit the evaluation team’s ability to make a definitive assessment of particular project elements for questions relating to this information, the team was able in some cases to use qualitative data as a proxy or substitute aggregate data. Sampling Strategy – While the evaluation team endeavored to select a randomized sample of respondents from each category, in some cases, AYP staff were called upon to facilitate contacts on behalf of the evaluation team. It is possible that this biased the respondents’ views or ability to speak freely with the evaluation team. This was mitigated to the greatest extent possible by: 1) contacting as many respondents as possible directly; and 2) in cases where facilitation was required, explaining to the respondent in detail how the evaluation team was independent of the AYP staff and sought frank assessment of their experiences. Scope – This performance evaluation was conducted over a 10-week period, which allowed the evaluation team only a snapshot of the vast range of AYP’s implementation sites and activities. 5 Incentivizing Respondents’ Participation – In order to encourage participation, the evaluation team provided light refreshments for the respondents, which could have influenced the feedback. In order to minimize the impact of this, refreshments were distributed at the end of the discussion. FINDINGS AND CONCLUSIONS Question 1: Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? Institutional capacity development of the MOE was a strong focus of the project’s work, both at the national and county/district levels. AYP complemented the institutional capacity development training with an annual institutional capacity assessment, which collaborated with the Ministry to conduct a self￾assessment of their capacity to implement ABE. The evaluation team had insufficient quantitative evidence to determine the impact of this work on the Ministry’s capacity to plan, budget, deliver, and measure ABE for youth. However, qualitative feedback suggests the AYP model built the MOE’s professional capacity to plan, deliver, and measure ABE; however, the evaluation found no evidence that the Ministry benefitted from budgetary capacity building. In addition, the sustainability of this work is limited by the MOE’s high turnover rates and the apparent lack of political will to institutionalize ABE service provision. AYP provided a number of other valuable assets in support of ABE which may be sustained into the future, including a teacher-training program at Stella Maris Polytechnic College and the installation of solar-powered lights in nearly all of the 147 MOE schools in which AYP is implemented.  To what extent were the Government’s expectations and policy needs met? Overall, the MOE was satisfied with the capacity building support it received, including the AYP curriculum and training at all levels. Ministry officials interviewed for this evaluation all indicated that AYP adopted an inclusive approach to implementation, including them in every activity and decision￾making process, and that the experience had been beneficial to their development. In some cases, MOE reported that they did not feel included in the project development; however, this may be related to high turnover within the Ministry staff, and those individuals were not present during this process.  How likely will the Government be to use the ABE materials in the future? The MOE is not providing accreditation of ABE programs implemented by other service providers nor certification for ABE learners at this time; they are currently operating their own ABE programs in a total of 19 sites. NGO service providers like NAEAL and LIYONET are also providing ABE classes through AYP, as well as other donors and NGOs (e.g., CONCERN, Save the Children). Going forward, the European Union (EU) indicated plans to pick up the AYP curriculum for a forthcoming ALP project in the south-eastern counties.  How are communities proposing to sustain the efforts financially or in kind? According to AYP’s Transition Plan document, the current strategy for continuing AYP in the absence of USAID funding is to transition to a wholly community-led model. However, the communities expressed apprehension and doubt about their ability to take on the financial responsibility for ABE provision. While beneficiaries acknowledged the value of the assets they had gained through AYP (e.g., solar lights, 6 trained facilitators, curricula, and revenue generation potential from skills training activities), they had serious concerns about their ability to pay facilitator and administrator stipends, and assume the cost of reproducing materials. Question 2: Youth Targeting  What strategies worked best for recruiting and retaining out of school youth? The most effective strategies for recruiting out of school youth to attend ABE classes included community publicity campaigns; and pro-active ABE committees, facilitators, administrators and learners interacting with youth encouraging them to enroll. The most effective strategies for retaining learners were pro-active ABE committees, facilitators, administrators and learners personally reaching out to youth and motivating them to return to classes. However, the most effective strategy to both recruit and retain out of school youth were the short-term skills and long-term livelihood trainings. There are many reasons why learners do not attend ABE classes regularly or drop out of the classes completely: they have conflicting work commitments, take care of their families or cannot commit for nine months to complete an ABE course level. Some learners also simply want to be able to write their name, and after they are able to do, they don’t continue with the classes. Thus, there are broader contextual issues that make learners leave the AYP classes; therefore, even the most successful strategies that AYP employs are not fully adequate to address the retention of its learners. Nonetheless, in the remaining time of AYP, another round of trainings should be provided to ABE committee and youth club members, facilitators and administrators on successful retention strategies. Future ABE programs need to establish pathways for a larger number of learners to get short-term skills and long￾term livelihoods trainings so they remain in ABE classes.  What interventions were most successful in helping to create positive outcomes? The main factors that contributed to a learner completing the ABE level he/she was enrolled in included: personal commitment, regular class attendance, engagement in classroom activities, support from families and communities, capable and engaged facilitators and a student-centric curriculum. Some learners just wanted to learn to write their name and gain some basic literacy and numeracy skills. The most dominant factor, however, was once again the offer of short-term skills or long term livelihoods trainings. Further, being an active member of the youth clubs and selling soap or garden produce was also a big incentive for learners to complete their level. Any future ABE project that aims for learners to complete an ABE level should provide free (or very low cost) learning opportunities; have capable and engaged facilitators and administrators; and have ABE committees that support the schools and learners. Further, the integrated model of AYP encompassing ABE and skills and livelihoods trainings is optimal, but significantly more pathways are needed for trainings and job connections to aid in learners completing an ABE course level. Question 3: Outcomes  To what extent did AYP link youth to “relevant and realistic” livelihoods training? The AYP model of providing livelihood trainings is effective and is the most in-demand area of their service provision. Most participants had an opportunity to learn skills that were identified to be in demand in the market and relevant to their needs through a youth-led Labor Market Assessment. Short-term skills training included: soap making, bee-keeping, food preservation, candle making, snail 7 raising, pastry-making, etc. Some 250 selected Level 3 learners had further opportunities to engage in long-term skills training in areas such as tailoring, carpentry, auto-mechanics, masonry, small engine repair, electricity and plumbing etc. A very small subset of 75 learners also had an opportunity to participate in three-month internships which were implemented as a pilot project in the fourth year of the activity. In addition, nearly all of the 147 AYP sites have a garden that is managed by the youth clubs, which provides some income for the clubs and also trains youth on agricultural skills, which is the highest sector employer in the Liberian economy. In an expansion of this work, 18 AYP sites implemented Grow Your Business, providing training for learners to expand their agricultural knowledge beyond traditional subsistence level to commercial-based farming, including marketing and financial management of agricultural endeavors.  To what extent did livelihood trainings lead to quality employment for youth? Available data on the number of AYP graduates who gained employment were outdated. AYP is reportedly undertaking a tracer study to provide more current and comprehensive information. Additionally, learners were provided with internships and linked with key service providers, but outcome level data on how frequently this led to gainful employment was also not available. According to the qualitative feedback in the field, most who had gained employment reported being self￾employed rather than attaining formal employment. On the other hand, there are individual-level cases of learners whose participation in AYP supported their transition to gainful employment, but they seemed to be limited.  To what extent did learners improve literacy, numeracy, and work readiness skills? The 2012/2013 OLA and EGMA assessments showed low, albeit statistically significant, gains in learners’ literacy and numeracy scores. After these assessments, AYP made significant revisions to the Level 1 and 2 curricula and placed more emphasis on facilitator training. The forthcoming OLA and EGMA endline assessments (baselines were done in 2015) will thus be more informative about how effective AYP is in teaching literacy and numeracy. Nonetheless, qualitative data gathered from the field visits clearly suggests that both male and female learners can significantly improve their literacy and numeracy skills if they are motivated, attend class regularly, participate in activities, and have competent and engaged facilitators. Any future ABE project needs to have standardized learning assessment tools and methods. Thus AYP needs to provide information, options, and standards to ABE committees on how learning can be assessed in the future. Further, the MOE also needs to establish learner assessment standards and tools to determine literacy and numeracy competencies that will align with their ABE curriculum. Question 4: Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? In responding to this evaluation question, it is essential to reiterate that there are gaps in the overall information available to the evaluation team, in order to identify the most effective AYP interventions 8 upon which to build. However, based on the information available for this performance evaluation, early indications suggest the following interventions were the most effective:  The literacy and numeracy curriculum provided by AYP seem to be effective learning tools to address learners’ needs, pending final OLA/EGMA results.  Short-term and long-term skills trainings aided in livelihood skills development and was also the most effective tool in recruiting and retaining learners.  AYP developed effective strategies to reach female participants and to meet the needs of learners in rural communities.  While it poses a challenge, the strategy of working through the MOE to implement ABE activities, rather than going solely through local NGO service providers, is crucial in building sustainability and scalability into project implementation strategies. 9 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE Social Impact (SI) is an international development consulting firm whose mission is to create dramatic improvements in the performance of organizations and programs working to enhance the social and economic well-being of people around the world. SI is contracted by USAID/Liberia to implement the Liberia Strategic Analysis (LSA) Activity, a 5-year contract to provide support to USAID/Liberia’s Program and Project Development (PPD) and Development Objective (DO) teams with project design, performance monitoring and evaluation, strategic planning and collaboration, learning and adapting (CLA) support. Through LSA, Social Impact was tasked to conduct a performance evaluation of the Advancing Youth Project (AYP). AYP, awarded through the EQUIP3 mechanism, is a five-year, $39.8 million activity implemented by Education Development Center Inc. (EDC) in six counties throughout Liberia. AYP works in close collaboration with the Liberian Ministry of Education to increase access to quality alternative basic education (ABE) classes, social and leadership development opportunities, and livelihood skills training for out-of-school youth ages 15 – 35, with no or marginal literacy and numeracy skills. At the time of this report (May 2016), AYP was scheduled to end in October 2016, with the possibility of a no-cost extension through June 2017. This performance evaluation of AYP examines the efficacy and sustainability of the activity and documents lessons learned through the implementation process in order to inform the design of future education projects that focus on USAID’s Education Strategy – Goals 1 (literacy) and 3 (access). This evaluation provides an independent examination of the overall progress and achievements of AYP in Liberia. It examines the activity from inception through the midway point of Year 5, including issues of sustainability, youth targeting, and outcomes. It also identifies achievements, performance issues, and constraints related to activity implementation. As AYP is midway through its fifth, and possibly final, year of implementation, this performance evaluation report also seeks to provide specific findings, corresponding conclusions, and concise, actionable recommendations about which component(s) of AYP could be scaled up, modified, or re￾designed to maximize results and impact for the remaining implementation period. These findings, conclusions and recommendations were shared and discussed with USAID/Liberia, AYP implementing partners, and the Liberia MOE, in order to increase transparency and ownership of findings, as well as to inform the trajectory of activity implementation for the final period of AYP’s lifecycle. EVALUATION QUESTIONS The following questions are the key points on which the evaluation was focused, in the design of instruments, analysis tools, and in the overall findings, conclusions, and recommendations of the evaluation reporting. 10 Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? o To what extent were the Government’s expectations and policy needs met with AYP? o How likely will the Government be to use the ABE materials in the future (confirm that the Ministry has provided accreditation and certification for the alternate learning program and ABE)? o How are communities proposing to sustain the efforts financially or in kind? Youth targeting  What strategies worked best for recruiting and retaining out of school youth?  What interventions were most successful in helping to create positive outcomes (i.e., completion of course-level 1-3, internship/employment, other livelihood interventions)? Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training? o To what extent did livelihood trainings lead to quality employment for youth?  To what extent did learners improve their literacy, numeracy, and work readiness skills to date? This will be informed by EDC's own Out-of-School Literacy Assessment (OLA) and the Early Grade Mathematics Assessment (EGMA), but an independent review of these materials will help inform the Mission and the GOL to become more effective at improving learning outcomes for young adults in the classroom. Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions (based on what has been learned from the accomplishments and challenges of AYP up until now) that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? The original Statement of Work for this performance evaluation included an additional question addressing the issue of cost-effectiveness. However, based on feedback from the implementing partner that there was insufficient data to fully answer this question at the time of the evaluation, and upon their advice that they were planning to undertake this examination during the remaining period of implementation, USAID approved its removal from the scope of this performance evaluation. The original scope of work can be found in Annex I: Evaluation Statement of Work. 11 PROJECT BACKGROUND Youth in post-conflict Liberia face a variety of challenges, including limited access to quality education. Fourteen years of civil war (between 1989 and 2003) devastated the nation’s economic and education systems, which limited young people’s ability to attend school regularly or at all. Thus, many youth lack basic literacy and numeracy skills: according to the World Bank EdStats, the literacy rate among youth ages 15-24 in 2007 (the only year for which data is available) was 49.1 percent, and among females, a staggering 37.2 percent (as compared to 70.8 percent overall and 65.8 percent for females in the Sub￾Saharan region as a whole). Such limited educational attainment in turn impacts youth’s employment and livelihood development potential, which is an extreme challenge to the country’s economic and social security. In response to these challenges, USAID developed AYP to work in conjunction with the Liberian Ministry of Education (MOE) to develop and pilot an Alternative Basic Education (ABE) curriculum that offers youth the knowledge and skills they lack. Working largely through MOE schools, teachers, and systems, the program offers literacy, numeracy, work readiness, health, livelihoods, and other life skills programming for learners with zero or extremely low literacy levels. In support of this effort, AYP also facilitates the organization of youth clubs, which provide social and educational opportunities for AYP learners, as well as peer support for strengthening academic and livelihood skills. In some cases, AYP also facilitates access to vocational and applied skills training that enhances learners’ capacity to find employment in formal or informal markets, or establish entrepreneurial endeavors. AYP continues the work started under USAID’s Core Education Skills for Liberian Youth (CESLY) project which developed and piloted the first level of the ABE curriculum. There was a limited gap in the service provision between the two projects, which enabled AYP to continue CESLY’s work with little interruption, expanding the curriculum to Levels 2 and 3 and reaching more than 22,000 additional direct beneficiaries with the literacy, numeracy, life skills, and livelihood skills training components of the project. Under the AYP Results Framework, the project has two primary strategic objectives: Strategic Objective 1: Capacity and Sustainability. MOE and CBOs have the capacity to support and manage alternative basic education programs in Liberia with efficiency, effectiveness, and equality. Strategic Objective 2: Access and Quality. Youth have access to clearly defined and relevant education and livelihoods pathways. Strategic Objective 1 has one identified Result: Increased Institutional Capacity to plan, budget, deliver, and measure ABE for youth. This is indicated by both MOE’s and local NGO service providers’ capacity to provide quality ABE, as well as the provision of a comprehensive ABE curriculum. Under Strategic Objective 2, there are two defined Results: 1) Increased access to relevant basic education and life, work, and technical skills training for out-of-school youth; and 2) Increased access to sustainable livelihood pathways for targeted youth. Result 1 is indicated by effective recruitment and retention of youth, especially women; the development of capable ABE facilitators; and improved literacy, numeracy, work readiness, and life skills for youth. Result 2 is indicated by the establishment of national and county-level alliances for youth employment; development of NGOs’ capacity to deliver livelihoods training; and linking youth (especially women) to relevant and realistic livelihoods skills opportunities. The full Results Framework for the AYP project is provided in Figure 1 below: 12 Figure 1: Advancing Youth Project Results Framework Result 3: Increased access to sustainable livelihood pathways for targeted youth Result 2: Increased access to relevant basic education and life, work, and technical skills training for out-of-school youth Result 1: Increased institutional capacity to plan, budget, deliver and measure Alternative Basic Education for youth Sub-result 3.3 Young people (especially women) are linked to relevant and realistic livelihoods skills opportunities Sub-result 3.1 National and county-level alliances for youth employment are established to provide work-based learning, skills training and entrepreneurship opportunities for targeted youth Sub-result 2.1 Strategies for recruiting and retaining youth, especially young women, are planned and implemented Sub-result 2.3 ABE students have improved literacy, numeracy, work readiness, health and life skills Sub-result 2.2 ABE facilitators (both government and non government) have the capacity to effectively teach all levels of the ABE curriculum Sub-result 1.3 A comprehensive ABE curriculum is developed and implemented Sub-result 1.2 Selected local institutions will have the capacity to partner with the MOE to deliver quality ABE instruction, and/or promote entrance and persistence in ABE Sub-result 1.1 MOE is able to set national ABE policies, prepare annual implementation plans and budgets, and train, support and monitor ABE teachers, based on evidence of best practices Sub-result 3.2 Selected NGOs have the capacity to deliver livelihoods training programs, based on assessed local market needs Strategic Objective 1: Capacity and Sustainability: MOE and CBOs have the capacity to support and manage alternative basic education programs in Liberia with efficiency, effectiveness, and equity Strategic Objective 2: Access and Quality: Youth have access to clearly defined and relevant education and livelihoods pathways Project Goal: Enhance the capacity of governmental and nongovernmental institutions to provide increased access to quality alternative basic education services, social and leadership development and livelihoods for youth and young adults, ages 13-35 who are un-schooled or out of school and have no or marginal literacy and numeracy skills 13 AYP PROJECT OVERVIEW AYP provides ABE programming to Liberian youth who never received any formal schooling or dropped out of school in early primary grades. The activity is implemented with Basic Education funds from the USAID/Liberia Education Office, with a portion of funding coming from the Economic Growth office to support vocational skills training, and a portion from the Health Office to support healthy lifestyles curriculum. This section provides a brief overview of the AYP activity; additional detail can be found in Annex VI: AYP Activity Overview. The basic education learning component of AYP supports out of school youth to improve their literacy, numeracy, work readiness and life skills through the provision of ABE. The program is open to all youth within the age range of 13-35 who reside in or near the communities where the AYP schools are located. The activity data shows that since its inception more than 22,000 youth have enrolled in the ABE classes, the majority of which are women. There is also a high dropout rate: out of the 22,256 enrolled, only 3,103 completed the level for which they had enrolled. Table 1: Number of Beneficiaries by Category, Achievement Level, and Gender Indicator Aggregate # % Male % Female Learners Enrolled 22,256 42 58 Learners Completed the Level for which Enrolled 3,103 36 64 Learners Employed 136 30 70 Administrators Trained 289 89 11 Facilitators Trained 983 83 17 Curriculum AYP comprises a three-level ABE curriculum, which is directly aligned with the MOE public school curriculum for primary grades 1-6. The ABE curriculum is also supplemented by interactive audio instruction (IAI) for Level 1, and mLearning component through mobile phones for Levels 2 and 3. Table 3 details the three levels, what subjects are covered, and their equivalencies in the conventional school system. Table 2: AYP Classes and Subjects by Level Level Subjects Classes Equivalency Level 1 Literacy, Numeracy, Work Readiness and Life Skills --3 ½ hours per class held four days a weeka --2 semesters spread over 9 to 10 months --Classes usually start in September --Schedule generally aligned with the MOE academic calendar Successful completers achieve 2nd grade equivalency Level 2 Literacy for Work, Numeracy and Life Skills Successful completers achieve 4th grade equivalency Level 3 --English, Math, Social Studies, and Science --Grow Your Business in 18 AYP schools Successful completers achieve 6th grade equivalency a This was the schedule at the time of the evaluation, which was condensed to mitigate implementation delays as result of the Ebola crisis; the original schedule was 3 days/week for 10 months. Learners are assigned to one of the three levels of the ABE classes based on initial placement tests, which are administered by AYP before the classes start, and given a completion exam at the end of the 14 academic year to test their mastery of concepts. AYP also uses two tools to measure learners’ gains in literacy and numeracy: The Out-of-school Literacy Assessment (OLA) and the Early Grade Math Assessment (EGMA). Administrators & Classroom Facilitators Each AYP school has one administrator who is often the principal of the MOE day school, as well as five facilitators for each site, who are teachers at the MOE day school (two for Levels 1 and 3, and one for Level 2). Facilitators and administrators are paid a monthly stipend by AYP for the duration of the ABE classes, in addition to their regular MOE salaries. All administrators and facilitators receive an initial 10- day training course and complete pre-tests and post-tests. Based on the results of these tests and their performance during the trainings, facilitators are assigned to teach one of the three levels. AYP has also formed a partnership with Stella Maris Polytechnic University to offer a pilot credit course (with a course curriculum, assessment and evaluation tools and a corresponding resource book) for teaching ABE, in which some AYP facilitators have been trained. Groups Associated with AYP Schools Youth Clubs – Learners at each site form youth clubs, which include an elected president, vice￾president, secretary and treasurer. Youth clubs play a role developing networks among learners, conduct other activities such as community clean-up, aid in recruitment and retention of learners for the ABE classes, and mobilize the establishment of small gardens and organize fundraising from short-term skills training activities such as soap making. ABE Committee – Each site is overseen by an ABE Committee (similar to a PTA) including an elected chairperson, co-chairperson, secretary, recruitment coordinator and advisor. It is made up of community leaders and elders, as well as the school administrator and youth club president. It oversees the implementation of ABE classes, ensuring that classes are held as planned, start on time, that facilitators are performing their duties as assigned, and that the classroom facilities are maintained. They also play an active role in recruitment and retention of learners. County Alliance – This group consists of county level officials from the MOE and Ministry of Youth and Sports (MOYS), NGOs and CBOs leaders, religious leaders, business owners and other leading community members. Its main role is to develop external support for youth development, including public-private partnerships to raise funds and facilitate applied learning opportunities for participants. NGO Partners – AYP has two other partners in the provision of ABE to Liberian youth. The Liberian Youth Network (LIYONET) is using the AYP curriculum in five school sites in Nimba County, with facilitators who are also MOE school teachers. The National Adult Education Association of Liberia (NAEAL) is offering the AYP Level 1 curriculum in 20 schools in two counties, however, they are using their own facilitators. NAEAL also has its own pre-Level 1 ABE curriculum that it offers to adults across the county, which has been used as a feeder for AYP learners. Livelihood Skills Training AYP emphasizes improving learners’ relevant work readiness skills and facilitating opportunities to improve their livelihoods. Most learners have opportunities to engage in short-term skills trainings in activities such as soap making, baking, beekeeping etc. A smaller subset of youth received longer-term vocational trainings in electrical and mechanical skills, masonry, carpentry, welding etc., which were implemented through the Liberia Opportunities Industrialization Centers that are located in two 15 counties: Grand Bassa and Montserrado. In complement to the LOIC training, about 250 learners had an opportunity to engage in 3 month internships at garages, tailoring shops, etc. to apply the skills they were learning in the LOIC. Finally, a pilot in-kind internship activity was implemented in the past year, through which some 75 learners had an additional opportunity to undertake an internship. Through these activities, AYP provided over one million person hours of training in workforce development through a combination of classroom teaching of work readiness skills, business skills and agriculture￾related skills, as well as short-term skills training. 16 EVALUATION METHODS & LIMITATIONS EVALUATION METHODOLOGY The AYP performance evaluation team consisted of three members: an international Team Leader/Senior Education Specialist; an international Evaluation Specialist; and one Liberian Education Context Specialist. The team was supported by another LSA Evaluation Specialist, and the LSA Chief of Party (COP) provided oversight of the activity, including review and approval of all final deliverables. The original design of this performance evaluation planned to undertake a mixed methods approach, combining qualitative data with analysis of quantitative data that had been collected by the AYP team during the first four years of implementation. However, the evaluation team had limited access to AYP’s quantitative data; thus, the evaluation relies heavily on qualitative information that is supported by quantitative evidence wherever possible. Document Review During the first week of the evaluation, the team conducted a thorough review of background documents related to AYP. A full list of the documents reviewed is available in Annex II: Evaluation Methods and Limitations. Field Work After the background review phase, the team spent four weeks in the field conducting site visits and collecting qualitative data in order to gain a deeper understanding of AYP, the experiences of its direct beneficiaries, and the local implementation environment. The school sites visited by the evaluation team were selected through random sampling of a full list of all locations within the three target counties. Table 3 lists the sites visited by county. Table 3. Field Sites Visited County Site Classification Montserrado County Grey D. Allison Elementary School Urban Kortu Town Community School Rural St. Paul’s Bridge Elementary School Urban Logan Town Central Public School Urban Nimba County Sanniquellie Elementary School Urban Gehwee Elementary School Rural New Yekepa Elementary School Rural Tondin Elementary School Rural Grand Bassa County Lowerharlandsville Public School Urban Little Bassa Public School Rural Benson River Public School Urban Cotton Tree Public School Urban 17 Key Informant Interviews During the site visits, the evaluation team conducted a wide range of Key Informant Interviews (KIIs) with AYP project beneficiaries and stakeholders. These interviews were guided by semi-structured interview protocols that identified the evaluation questions that needed to be answered through the interview and identified probes to follow up on information related to questions (see Annex III: Data Collection Instruments, for a copy of the interview guides). Overall, respondents associated with the learning sites were selected and identified based on their affiliation with the selected site. For a vast majority of the schools visited, the evaluation team interviewed most of the individuals associated with the site: the school administrator, classroom facilitators, ABE Committee members, and a majority of the learners. Table 4 below enumerates the number of KIIs targeted and conducted with each type of stakeholder group; a more detailed breakdown of these categories is included in Annex II: Evaluation Methods and Limitations. Table 4. Number of Key Informant Interviews Targeted and Reached Respondent Group Total targeted Total Completed AYP Head Office Staff 10 10 AYP partners 6 4 Government of Liberia 6 8 Other donor programs and USAID offices 5 2 ABE Leaders and Community Partners 18 19 ABE Learners 9 3* Employed and unemployed graduates of AYP 6 9 * Planned individual interviews with AYP learners were replaced with focus groups in response to field contexts Focus Group Discussions The evaluation team also conducted focus group discussions (FGDs) with select groups of AYP stakeholders focusing on one or many of the evaluation questions. For the ABE Committees, County Alliance and Youth Club members, group sizes were smaller – between 3-5 respondents per group. However, the Learner focus groups were much larger – the average size was 16 participants. The discussions were also organized around a semi-structured focus group guide (see Annex III: Data Collection Instruments, for a copy of the focus group guides). Table 5 below lists the respondent groups that participated in focus groups, along with the original target and the completed sample size per respondent category. Table 5. Number of Focus Groups Targeted and Reached Respondent Group Total targeted Total Completed ABE Committees 3 4 County Alliance members 3 2 Youth Club members 3 2 ABE Learners 3 9* * Planned individual interviews with AYP learners were replaced with focus groups in response to field contexts 18 Data Analysis and Reporting Data collected during KIIs and FGDs were captured in note form, typed up, and circulated to other team members for additional inputs. The final versions were processed systematically by sorting observations according to a coding system developed by each researcher for their specific topic area. These data were analyzed by the team using an inductive approach, gathering all of the data before engaging in a day-long group discussion held to identify and debate amongst the evaluation team members the themes and patterns identified within the data. This process informed development of key findings, conclusions and recommendations that make up the bulk of this report. DATA LIMITATIONS Limited Access to Data – As mentioned above, the evaluation team had limited access to AYP’s learner data and other information that the project deemed too sensitive to release to an external evaluation team. While the team was able in some cases to use qualitative data as a proxy or to use top￾line reports with aggregate data to assess the efficacy of project activities, it did limit the evaluation team’s ability to make a definitive assessment of particular project elements (specific details on limited data accessibility are included per relevant finding). In addition, it was difficult to ascertain exactly what are the motivations and political will of the MOE to institutionalize ABE into their strategic plan. Although a number of Ministry officials were interviewed during the qualitative data gathering process, many times their answers were vague or contradicted each other. Thus, the evaluation could not take into account any future planning for ABE by the MOE and took the current status at face value in making recommendations for AYP. Sampling Strategy – While the evaluation team endeavored to select a randomized sample of respondents from each category, in some cases, AYP staff were called upon to facilitate contacts on behalf of the evaluation team. It is possible that this biased the respondents’ views or ability to speak freely with the evaluation team. Scope and Timeframe of the Evaluation – This performance evaluation was conducted over a 10- week period, which allowed for two days of background document review, which limited the team’s capacity to review the volume of project documentation that was available. In addition, the four weeks allotted to field work allowed the team to visit only 12 of the 147 AYP sites, and in three of the six counties in which AYP operates. This gave the evaluation team a snapshot of the vast range of sites in which the project is being implemented. Incentivizing Respondents’ Participation – The evaluation team experienced some challenges in hosting focus groups and ensuring that respondents would attend without any incentives to compensate them. While no monetary incentives were offered, the evaluation team did provide light refreshments for the respondents, which could have influenced the respondents’ feedback. In order to minimize the impact of this tendency, the refreshments were distributed at the end of the discussion period. Additional detail on the evaluation’s overall approach to the methodology and potential limitations of the data is available in Annex II: Evaluation Methods and Limitations. 19 FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS Question 1: Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? As the first Strategic Objective for AYP, institutional capacity development of the MOE was a strong focus of the project’s work, especially in the first few years of implementation. Based on Sub-result 1 of the Results Framework, AYP aimed to develop the MOE’s capacity to “set national ABE policies, prepare annual implementation plans and budgets, and train, support, and monitor ABE teachers, based on evidence of best practices.” In pursuit of this goal, AYP instituted a series of trainings and capacity building efforts within the Ministry, initially working as embedded staff in the Ministry itself, and later from the Monrovia-based AYP office. Training focused strongly on the county and district level staff, developing the capacity of ABE Supervisors, the County Education Officers (CEO) and District Education Officers (DEO), and their team members (M&E, finance, etc.). Training sought to familiarize these stakeholders with the ABE curriculum, its reporting, management and leadership, and also to develop their computer skills in order to streamline and optimize the Ministry’s processes, especially for reporting. AYP complemented the institutional capacity development training with an annual institutional capacity assessment, which collaborated with the Ministry at both the national and county levels to conduct a self-assessment of the Ministry’s current capacity to implement ABE. This process has been undertaken three times to date. As a result of this assessment, AYP developed an annual workplan for the MOE, which included some 25 activities that the Ministry would implement throughout the year in order to expand its capacity in ABE service provision. This process was timed to coincide with the Ministry’s annual planning processes so that the recommendations could be incorporated into their annual plan. As the evaluation team did not have access to the MOE’s institutional capacity assessments or to the top-line reports that were developed for USAID that detailed the results of the ICAs, and there are no outcome level data reported in PIDS for this indicator, there is insufficient evidence to quantify the impact of this work on the Ministry’s capacity to plan, budget, deliver, and measure ABE for youth. Based on qualitative feedback from the Ministry staff interviewed for this evaluation, the AYP model built professional capacity within the Central office staff, CEOs, DEOs, MOE teachers, etc., to plan, deliver, and measure ABE through the trainings, development of ICAs and Program Quality Standards (PQSs), and joint monitoring visits. In addition, information reported in the AYP quarterly and annual The Ministry of Education ran an accelerated learning program after the war – it was six years of basic education programming condensed into three, and then a transition to mainstream schools. The need was so huge, but there was not the capacity to meet both the wide range of learners’ ages and the scope of the need within the Ministry’s resources. - MOE Deputy Minister 20 reports offered some insight into the Ministry’s self-assessment of capacity, including strengths and weaknesses of the institution at both national and county levels. The Quarter 4, Year 4 Quarterly Report (which was the most recent report available) identified Financial and Human Resources Management as the two areas in which the MOE’s capacity is strongest, largely due to the fact that centralized systems and policies support these areas, so that the MOE staff have clear guidance and knowledge in these areas. Conversely, the report identified Program Implementation and Management, and Program Performance Management as the two areas in which the MOE still has a need for further capacity development. These included areas such as: opening additional ABE sites which would also include livelihoods training; adopting an integrated approach to ABE service delivery; establishing a policy framework for training MOE teachers to offer ABE; establishing linkages and networks for ABE; providing accreditation and registration for ABE service providers; and enhanced data capturing and reporting systems. Enhanced development of the MOE’s capacity in these areas was anticipated through the adoption of the PQS (which are discussed in more detail below) as well as through “problem solving, training and on the job learning at the site level.”1 Overall, qualitative data captured through the evaluation interviews indicated that capacity development training for the MOE was well-received: all individuals interviewed were very complimentary about the quality and utility of the AYP training and its impact on their capacity to provide quality ABE for their constituencies. However, the sustainability of this work was limited by the fact that the Ministry has high turnover among the Executive level staff, and frequently reallocates central office and local staff to other locations or assignments; thus, teams whose capacity was developed to start implementing upgraded systems and procedures were broken up and staff members reassigned to other places or teams. While the knowledge and skills of the beneficiaries presumably remains, respondents had limited expectation that these enhanced systems could be sustained in the absence of an enabling environment. As the overall capacity of the Ministry’s systems are limited, they felt it was unlikely that an individual who has had limited exposure to computerized reporting systems, for example, would be able to push this model forward in a new environment without the proper tools and guidance to institutionalize them. MOE officials interviewed for this evaluation could not offer specific examples of how the Ministry had strengthened its budgetary capacity through AYP. When the team inquired about this specifically, respondents did not recall any specific intervention addressing budgetary guidance in the AYP training. It is possible that budgeting guidance may have been included in the ICA workplans, but it was not remembered or could not be directly implemented due to external factors. The biggest limitation on the sustainability of AYP’s work with the MOE is the competing priorities of quality versus access; the MOE Central office indicated that, given the limited budget allocated for education, their first priority is to provide higher quality education for conventional school students, and thus adult education could not be their primary focus at this time. This dynamic will affect the program’s sustainability if the systems and tools provided and recommended by AYP are not taken up as part of the Ministry’s required systems. In some cases, motivated CEOs and DEOs encouraged greater focus on ABE by conducting standard monitoring visits to AYP sites and submitting reports to the Central Ministry as part of their regular reporting system. However, this was not a required part of their assigned duties and was done in part to encourage the Ministry’s attention and support for ABE classes by integrating it into the regular educational system. 1 AYP Quarter 4, Year 4 Quarterly Report, pg. 15. 21 The AYP team was well-aware of the challenges to sustainable capacity development within the Ministry and endeavored throughout the project lifecycle to address these issues with various strategies. When it became clear that the Executive level staff at the Ministry was not signaling a desire to institutionalize ABE within the MOE’s service provision, AYP established a Technical Working Group (TWG), comprised of ministry officials at the Assistant level, as well as representatives from other donor agencies and education-focused NGOs. The TWG identified nine PQS to guide quality implementation of ABE programs. Although the Ministry approved the PQS standards, according to AYP reports none had officially been launched at the time of writing this report. This was a necessary step in introducing them to ABE service providers and initiating training on them. In order to address this issue, AYP has started selecting service providers to test them, which will enable them to start familiarizing other ABE service providers with them before the project closes out. Finally, AYP reported that, as of the writing of this document, the MOE has indicated it will incorporate an element of Adult Education into its priority areas for the forthcoming three-year Education Sector Plan. While this is an encouraging step, it is too soon to know the full scope of this effort and whether it will be sufficient to institutionalize the Ministry’s provision of ABE into their standard education offerings. In addition to the MOE, the MOYS was engaged in limited capacity building efforts and strategic planning efforts. The MOYS’s support for informal vocational training programs and overall livelihood skills development was a desirable element for AYP to include in their service provision; however, the basic education mandate of the project limited the capacity of AYP to engage MOYS as a strong partner in their work. With the inclusion of cross-sectoral funding from the Economic Growth office, the project was able to engage them in some activities and take advantage of their value added to the range of activities offered to the learners. Curriculum and Service Provision During its lifecycle, AYP developed and tested three levels of ABE curriculum, which include textbooks, learner workbooks, and facilitator guides for each subject, and revised Levels 1 and 2 based on user feedback; Level 3 has not yet been revised. A learner who successfully completes Level 3 of this curriculum is considered to have achieved the same proficiency as a student in Grade 6 of the conventional Liberian education system. Overall, the AYP curriculum contributed significantly to the MOE’s ability to deliver quality ABE, in that it reportedly makes up the bulk of the Ministry’s official ABE curriculum, which is mandated to be used in the Ministry’s ABE classes. However, the MOE is currently operating its own ABE programs on a very limited scale: according to Ministry officials, ABE classes being implemented through the Ministry’s own volition (excluding other donor- and NGO-led ABE service provision) are offered in a total of 19 sites. In addition, the accelerated ABE curriculum is not being offered in the Ministry’s night school program (which is targeted for adult learners); these classes follow the standard day-school curriculum. Sub-result 1.2 of the AYP Results Framework focuses on building the capacity of “selected local institutions … to partner with the MOE to deliver quality ABE instruction, and/or promote entrance and persistence in ABE.” While the first evaluation question focuses specifically on the MOE’s capacity development, AYP also worked to develop the capacity of NGO partner organizations to implement the ABE curriculum in a variety of different settings and formats, including sub-grants to NAEAL and LIYONET to provide ABE using the AYP curriculum. 22 In the case of NAEAL, the organization has historically provided pre-Level 1 education for youth who have never been to school before; the AYP curriculum will be a natural expansion of their current work to subsequent educational levels and they plan to implement it exactly as its provided. NAEAL’s model works within community structures: they train local community members as facilitators and use community space for classes, rather than working through the MOE systems. For LIYONET, they are using MOE teachers and classrooms but have also integrated elements into the curriculum that are adapted from their other programs, such as play-based learning. Each of these models will provide compelling evidence of various implementation models in the coming year, as they expand their AYP-funded offerings and gather feedback on the impact of this work. The provision of ABE curriculum to these NGO partners was also accompanied by capacity building training, organizational capacity assessments (OCA) and one-on-one coaching and support as the partners implement ABE using AYP materials. These efforts not only expand the capacity of NGO partner organizations to provide ABE training, it also enhanced the sustainability of the project investment by expanding the range of providers who are equipped to implement the AYP curriculum after the project ends. Both service providers spoke highly of the support they received from AYP and provided specific examples of how their organizational capacity had been built through the collaboration, which included budgeting skills training. Going forward, it is expected that the European Union (EU) is going to pick up the AYP curriculum to use in a forthcoming project that will implement ABE training for a younger cohort of learners in the south-eastern counties of Liberia. In addition, other donors and NGOs are providing a variety of alternative education (AE) learning opportunities, including CONCERN, Save the Children, among others. Teacher Training Another element of AYP’s work to support stronger capacity of the MOE to provide quality ABE instruction is the development of a teacher training program and the training of facilitators to implement ABE. Initially, AYP implemented teacher training directly through a Master Trainer program. However, the current teacher training model works with Stella Maris Polytechnic College to facilitate a 2-week course for in-service MOE teachers to learn how to teach ABE classes. Thus far, they have worked with 50- 60 total beneficiaries, who are trained in curriculum, educational leadership, and other skills. AYP staff feel it would be the most beneficial model to transition this into a pre-service program, possibly even as a certificate program for aspiring teachers at Stella Maris and elsewhere. The program also wanted to work with the Rural Teacher Training Institutes (RTTIs) to take up the curriculum, but eligibility and accreditation policies of the MOE were not conducive to training of ABE facilitators. AYP did continue to work with the RTTIs through Technical Working Groups, both for development and revision of content, as well as more recently in the development of the introductory course that AYP completed with Stella Maris. The opportunity to work with AYP has been an educational and rewarding interaction for our organization within the thematic area of education and youth work. Staff are equipped with knowledge, financial management tools, budgeting skills. The knowledge that we gained through AYP can be used in future projects, even after AYP ends. Their interactive, collaborative mentorship gave us better experience to expand our partnerships with other, bigger organizations, and to meet their deliverables requirements. In the absence of AYP, we can still manage to continue. We will seek other donor funds but AYP is becoming a core element in our activities, so we’ll get other funds and sponsors. - NGO Implementing Partner 23 According to AYP staff, the use of MOE teachers to facilitate AYP classes was a conscious decision to enhance the sustainability of the investment beyond the project period. Overall, respondents felt that the primary benefit of using the MOE teachers was the enhanced sustainability of building Ministry teachers’ capacity to provide ABE – which would also enable the MOE to institutionalize ABE service provision when AYP ends. Another benefit of this model is that the skills and strategies taught in the AYP teacher training program carry over to the convention classroom – especially learner-centric teaching methods. While there are benefits to the strategy of working through MOE facilitators who are paid by USAID￾funded stipends, informants also identified some drawbacks. The most frequently mentioned issue is that the MOE Central Administration often reallocates their teachers to other schools, districts, or even counties, leaving the ABE classes without a trained facilitator. In these instances, the school administrator has to take over the teaching position until another facilitator can be developed. The Office of Basic and Secondary Education is responsible for transferring facilitators from one site to another, and AYP enjoyed a strong relationship with the Assistant Minister for Basic and Secondary Education which enabled the project to limit the transfer of AYP facilitators (or arrange for them to be moved to another AYP site). In addition, DEOs play a role in this process as well, and facilitators who are notified of impending transfer sometimes alert them and the AYP team of imminent transfer plans if they’re not scheduled to go to another AYP site, so AYP can request the MOE to change this decision. Overall, this process appeared to have worked well to limit the impact of this challenge but did not alleviate it altogether. In addition, the choice to embed outsiders in small, isolated communities must be weighed and balanced against other implementation models, such as training community members to facilitate ABE classes. One major barrier to retention of female learners is husband’s jealousy and mistrust over their wife’s choice to be outside of the home in the evenings. NAEAL has mitigated this issue by adopting a strategy of training a local community member (who is chosen by the ABE Committee) to facilitate ABE classes. Having a known and trusted individual teaching classes was felt to be more comfortable for the community overall and especially for addressing the issue of women being away from home to attend night classes. For more information on challenges and effective strategies for learner retention, please see the section addressing Question 2: Youth Targeting, below. Monitoring and Evaluation As part of the institutional capacity development (ICD) support provided to the MOE, AYP conducted joint monitoring visits with the CEOs and DEOs, to strengthen their capacity to monitor ABE classes on Yes, all our facilitators trained by AYP also teach regular school. The training from AYP can carry over to the regular classrooms – the strategies, the AYP literacy for work they use in the regular classroom. I went to AYP training, and it taught me how to teach others, how to talk to them, other strategies to teach them, so while you’re teaching yourself, you’re also learning. - ABE Committee Member, AYP School Taking an external facilitator out to a community may make it harder for the beneficiaries to open up to them. If you work through local people, and develop facilitators from the local communities, you can be more sure of their trust and acceptance. This can also alleviate some of the challenges of the husbands’ and families’ resistance to women going out in the evenings to study, because they know the other individuals involved in the class. - NAEAL, NGO Implementing Partner 24 their own. There was also some level of information sharing between AYP and the county and district￾level staff, but this does not appear to have been institutionalized and was to some extent dependent on the engagement level of the MOE staff to undertake and sustain this relationship. There were certainly individual county- and district-level staff who had worked closely with AYP staff and had undertaken regular reporting of ABE class work to the MOE Central Administration. However, respondents felt their capacity could have been further strengthened if AYP had shared tools and strategic planning information with the CEOs and DEOs. According to respondents interviewed for this evaluation, AYP’s Learning Resource Center (LRC) staff and the MOE County and District staff conducted joint monitoring visits but did not share the monitoring tools, and strategic planning information with them as part of the AYP routine. Therefore, MOE staff on the ground felt they had little information or contextual understanding about the project’s objectives, which would have been beneficial to them, as the CEOs and DEOs who are more engaged and supportive of ABE provision could collect additional information through their work that might be useful for the AYP administration. While AYP has undertaken a wide range of research studies and assessments, one area of interest that remains to be answered is the extent to which the AYP program is successful in building learners’ literacy and numeracy skills (which is discussed in more detail under Section 3: Outcomes and should be further illuminated in AYP’s forthcoming 2015/16 OLA and EGMA Endline Reports), and how that translates into improved livelihoods opportunities for the beneficiaries. During evaluation interviews, the MOE specifically requested a tracer study to determine to what extent ABE programs support adult learners to gain more secure livelihoods: “No tracer study has been done to examine how these people have benefitted from ABE programs. Are they really utilizing these skills to build their independence and to reduce their potential security risk on society? Without this understanding, we can’t be sure that the ABE education we’re providing is effective; we need this information.” Given the MOE’s limited resources and the vast scope of the challenge they are addressing within the educational system (among both school-aged and adult learners), the Ministry identified this lack of information as a barrier to institutionalizing this program. However, AYP indicated that a mini￾tracer study was planned for the final phase of the project, which may help to encourage the Ministry to prioritize ABE investments in the future. Solar Lighting As most of the AYP learners are adults supporting a family, and need to work, the AYP classes are offered in the late afternoons or evenings. This required AYP to facilitate lighting for the schools, as they did not have them. Through many permutations, they arrived at the strategy of installing solar lighting and training the local communities to perform routine maintenance on them. According to AYP staff, solar lighting systems have been installed in all but five AYP sites – in which there were structural issues that prevented installation, so they are working to identify another solution. In general, this system has worked well, and the evaluation team saw evidence in every location visited of functioning lighting (sometimes the only electric lighting available in the community). This provided a quality learning environment for the AYP beneficiaries and also supported overall development as some children in the community took advantage of the lighting to study at night for conventional classes. Going forward, this asset will require some type of sustained investment to be maintained. Even within the scope of AYP’s work and with the benefit of USAID’s financial support, there were sites in which the solar lights needed repair or were not working in some rooms. Whether ABE service provision is transferred to the communities, the MOE, or another donor-funded project, the ongoing maintenance 25 of the lights will be an issue to address in support of a functional classroom for adults who wish to learn at night. Photo 1: Solar panels at Sanniquellie Public School provide lighting for AYP classes  To what extent were the Government’s expectations and policy needs met with AYP? As evidenced in the above section, the MOE was satisfied with the capacity building support it received, including the AYP curriculum and training at all levels. Ministry officials interviewed for this evaluation all indicated that AYP adopted a very inclusive approach to implementation, including them in every activity and decision-making process, and that the experience had been very beneficial to their development. The main issue expressed by the MOE Central Administration representatives interviewed was that the project was developed and given to them without allowing the Ministry to have significant input into its development. While they accepted the project as it was provided, they did not really have an opportunity to develop expectations, as they were engaged in a very limited capacity on the strategic planning process that identified the need for AYP and its composition. However, it is important to note that other respondents testified that the MOE was heavily involved in the project design but that the individuals involved in that process have since left the Ministry. Thus, the respondents who reported a lack of engagement may simply not be aware of this history or may be reporting it based on other experiences that are being generalized to AYP. This indicates both that high turnover in the Ministry staff impacts the level of institutional memory available for information gathering as well as programmatic and policy consistency, and that a strong champion is needed who is not a political appointee and therefore has a greater chance of serving in the Ministry in the long term. This champion could then consistently back service provision of ABE.  How likely will the Government be to use the ABE materials in the future (confirm that the Ministry has provided accreditation and certification for the alternate learning program and ABE)? We should sit together to plan and implement projects – we want to be in the driver’s seat, but they [donors] have the greater capacity. - MOE Official 26 The MOE is not providing accreditation of ABE programs implemented by other service providers nor certification for ABE learners at this time; nor could the evaluation confirm with any level of certainty at what point in the future this may happen. AYP representatives indicated that the Ministry does not have an established system to provide accreditation and instead can offer licenses to service providers to implement ABE on its behalf: “The MOE doesn’t really have a process for accrediting ABE programs – they just give a permit to operate. We are working with the MOE to establish accreditation-type terminology – may become ‘registered’ or ‘licensed’ instead.” Respondents within the Ministry also indicated that they are “working on a plan” to provide accreditation. However, given the uncertainty about this process, it is not possible for the evaluation team to establish a timeline for this to be completed. At the current time, the MOE is also not providing certification for learners; AYP provides a completion certificate for learners who complete Level Three but there is not an established system in place within the MOE to enable AYP completers to transition to MOE schools without taking the Grade 7 entrance exam. In one case, a DEO said that he provides a waiver for any AYP learners who pass the Level 3 completion test to be immediately accepted into Grade 7 of the conventional school system, but this was at his discretion, so it is not possible to generalize this finding to other locations around the country. Learners interviewed for this performance evaluation certainly desired certification – as well as completion report cards for Levels 1 and 2, which they felt would be useful for them in accessing livelihoods and training opportunities outside of AYP. However, according to Ministry officials, the MOE has developed and adopted as policy an ABE curriculum, of which the bulk of the materials are adopted from the AYP curriculum. This curriculum is being used on a limited basis in the 19 MOE sites that offer ABE, as well as being provided to AYP’s NGO implementing partners, NAEAL and LIYONET. The MOE may be using it or have future plans to use it in ABE projects implemented with other donors, such as the anticipated EU project.  How are communities proposing to sustain the efforts financially or in kind? While communities displayed various states of readiness to undertake the provision of AYP on their own, the overall sense is that the communities themselves are not proposing to sustain the efforts so much as AYP is expecting them to assume responsibility for the project, and the communities themselves feel very apprehensive about their ability to do so. Most respondents had limited expectations for the community’s ability to assume implementation of the project, as illustrated by the range of responses in the text box below. We can go and talk to those in the community, to say “Look, this program is good. I know you know the importance of it. This guy has been trained, he is here – can he continue and can you support him?” These are the kind of things that we are planning to do but that will be on a very small scale. - County Alliance representative, Nimba The solar lights that were installed will still be here. We will continue to teach our people. Those trained in skills will still train others on how to do those skills. The club will continue with its soap making and gardening strategy to generate additional funds for the up-keep of the program, though may not be enough to compensate the facilitators to keep the classes running. - ABE Committee member, Nimba We are trying to make things better, we now have lights in the school, and we have some serious learners. We want to get together with the facilitator and then go talk to the MOE in Sanniquellie and ask them to continue the ABE classes. - Youth Club President, Nimba We need a different donor or the MoE to take over the AYP, otherwise ABE will go down. - Facilitator, AYP School, Montserrado 27 According to AYP’s Transition Plan document, the current strategy for continuing AYP in the absence of USAID funding is to transition to a wholly community-led model. In order to support this effort, the Transition Plan outlines a series of meetings and dialogues with the various project stakeholders, including MOE Central Administration, County, and District Offices; Learning site stakeholders (administrators, facilitators, ABE Committee and County Alliance members, learners and youth club members); and other community leaders (such as Town Chiefs), to encourage community support and buy-in for this transition process. As of the writing of this report, this process had only just begun and no community meetings had been undertaken yet. Nonetheless, the evaluation team found that most learning site stakeholders were aware of AYP’s impending closure, but few sites had specific plans about how to sustain the project relying solely on local funding. While beneficiaries acknowledged the value of the assets they had gained through AYP (such as the solar lights, the trained facilitators, the range of curricula available to them, and the potential for the skills training activities such as soap making to generate revenue), respondents also had serious concerns about the community’s ability to pay facilitator and administrator stipends, and assume the cost of reproducing materials. There seems to be little question that financial resources do exist which could offset some of the costs of ABE classes if the youth clubs were interested in doing so. Activities like soap making and gardening have generated revenues that most reports indicate are substantial; the evaluation team heard figures ranging from 17,000 – 25,000 Liberian dollars and even higher. However, when the youth club representatives were asked about these assets, they were extremely reluctant to disclose specific sums and seemed worried that they might be taken away from them, so the evaluation team tried to be sensitive in probing for specific information. Thus, the team was unable to gather a clear picture of exactly how much funding is available and which communities might be willing to use it to support ABE classes. Revenues are held by the youth clubs and are used at their discretion. In some cases, we heard they may be given out as micro-loans to support small business ventures, but this is at their discretion. In most cases, when we asked administrators and ABE Committee members if these funds could be used to support ABE costs, they were hesitant to commit to that, as they did not have the authority to make that decision. Finally, the question of communities’ capacity to sustain ABE in the absence of donor or Ministry support was very often approached with a strong sense of donor dependency. Many respondents expressed confidence and expectation that someone else would come along to pick up these costs on their behalf if USAID elects not to continue service provision. As donors in general, and USAID in specific, contribute significantly to Liberia’s economic development, many people have become accustomed to donor-driven systems and lack confidence in their own capacity, and the capacity of their government, to provide for them. Two respondents spoke passionately about the issue of donor dependency during interviews, and both identified community-based ownership of projects as the key to resolving this issue. If communities were empowered to establish diversified partnerships with governments, business communities, and other stakeholders, they could provide their own stable sources of income and enhance their own capacity. Some skills, livelihood activities have some funds on hand that we believe can also sustain the program when AYP isn’t here [but] it’s going to be difficult for us. Our government is not that strong in the area of funding for education; that’s why I’m trying to partner with other people outside [such as private sector partners and local NGO service providers for skills training], so if we can get some funding to help us take over, that would be highly appreciated, I must admit. - County Alliance member, Nimba County 28 Photos 2 & 3: Youth club-led income-generating activities include soap-making (left) and community gardens (right) Question 2: Youth Targeting  What strategies worked best for recruiting and retaining out of school youth? Recruitment After AYP commenced in 2011, public announcements were made by community and village chiefs and leaders, MOE at the national and county levels, and AYP staff. As ABE committees were formed and administrators and facilitators hired, they played a central role in spreading the word that free ABE classes would be offered at a MOE public school in the community, and interested youth should enroll. Subsequently, every year existing learners, facilitators, administrators, youth clubs, ABE committees, AYP staff, local MOE offices and community leaders publicized the ABE classes prior to enrollment periods. Youth took placement tests to determine their appropriate level, and AYP provided each school site a list of learners per level. The evaluation data demonstrated that the best strategies for recruiting out of school youth are:  Publicizing the classes through flyers, radio, text messages, megaphone announcements, village and community leaders etc.  ABE committees, facilitators, administrators, youth club members actively going door to door and telling youth about the classes in the community  Having existing learners act as spokespersons for the program with their friends, families and in their communities  Stressing that participation in the ABE classes is free  Informing potential learners that other complementary components of the program offer some short-term skills and long-term livelihood trainings 29 The ABE classes were accessible to any youth in the communities where the AYP schools were located. The main challenge to enrollment proved to be the opportunity costs faced by youth, and this was an even more crucial issue for retention. Retention One of the greatest challenges faced by AYP was the retention of its learners. After enrollment, learners’ numbers per level declined over the academic year. In many cases, AYP data reflected 30-50% of learners leaving the program after two or three months. The main reasons that learners left included:  Having to take care of children and families (mostly for women)  Being too tired after working during the day to be able to come to class in the evening  Having work responsibilities in competition with ABE classes  Not being able to attend classes during the farming and harvest seasons  Learners moving away from the community  Different expectations of learners from the classes o Many learners only wanted to learn to write their name, and once they were able to do so, lost interest.) Many other learners expected to learn a skill or get livelihood training immediately after enrollment without having to attend classes  False perceptions about the program (e.g. that learners will receive monetary benefits or employment  A nine months academic calendar commitment being too long  Lack of personal commitment  Living far from the AYP school Many of these reasons also served as barriers that kept youth from enrolling. AYP was adaptive to retention issues and held workshops with administrators, facilitators, ABE committees and youth clubs, engaged community leaders, and held meetings with community members. The successful retention strategies included:  Personal interaction: facilitators, administrators and ABE community members went to the houses of youth and encouraged them to return. Facilitators and administrators also acted as role models and explained how they came from difficult backgrounds but went to school and became teachers. This demonstrated how persevering with education has long term benefits. “We heard about AYP from the facilitators and ABE community members who came in the community to promote the program.” - Level 1, 2 and 3 Learners, AYP School “Retention is a big challenge as we are working with adults who come to classes with different motivations. Many are family people who need to spend time with their children in the evenings and some move from community to community often. In the farming season many move to work. Many other learners want to have life-skills training without going to classes.” - AYP ABE Coordinator “Many learners quit immediately as they want to get some sort of livelihoods training without doing any of the three levels. They just want to get some livelihoods options so they can make money.” - Male Facilitator, Level 2, AYP School 30  Peer to peer engagement: youth club leaders and other active learners visited, called, and/or texted youth encouraging them to return  Retention Officers: Pro-active retention officers in the ABE committees were dedicated staff who identified and followed up with youth who had dropped out of classes.  Gender targeting: females visited other female learners who had stopped coming to class instead of males (and vice versa) in their homes, as this was more culturally acceptable in the local communities,  Incentives: o Short-term skills and long-term livelihoods training o Village Savings and Loan Associations (VSLA) o Learners with children were encouraged that by learning to read and write, they would be able to help their children with school work and be a positive role model for them  Inclusion mechanisms tied to incentives: The ability to be part of active youth clubs linked learners to a form social support and accountability while facilitating useful income-generating activities such as growing crops in gardens or making soap and selling these products in the market.  Success stories: sharing stories of previous learners gave the learners confidence by linking literacy and numeracy skills to how a more whole education can improve their lives and lead to better paying jobs.  Partner approach: Husbands and wives were encouraged to attend the same class so they had security of being together.  Student-centric teaching: training the facilitators and administrators to make the classes student centric and having group learning activities allowed learners to be engaged and not get bored.2 A critical issue to highlight pertains to the provision of short-term skills and long-term livelihoods trainings. While being one of the most successful motivations for youth to participate in ABE classes, it created unrealistic expectations for youth who were unprepared to commit to attending class regularly, complete the necessary class work, and be active youth club members. Thus the skills and livelihoods trainings also paradoxically lead to some learners dropping out from the AYP classes; especially when these youth did not have immediate access to trainings.  What interventions were most successful in helping to create positive outcomes (i.e., completion of course-level 1-3, internship/employment, other livelihood interventions)? 2 This is an integral component of the ABE curriculum of AYP. The facilitators, ABE committee members and the female learners go in the community to encourage those who have not been regular in school to return. They usually tell them the importance of education. Women have more been successful in bringing back their colleagues to school and retaining them as well. - Administrator, rural AYP School 31 AYP ABE Curriculum Technical and pedagogy experts developed the AYP curriculum, and after an internal review it was sent to the TWG. The TWG then recommended changes which were incorporated. The curriculum was then field tested in the AYP schools, and AYP staff obtained feedback from class observations and interviews with facilitators, administrators and DEOs and CEOs. Subsequently, all issues regarding the curriculum’s implementation were discussed, and the curriculum for Levels 1 and 2 was revised and finalized. AYP plans to provide information for MOE to revise the Level 3 curriculum (if needed in the future). Extensive thought and effort was put into developing the AYP curriculum to not only align it with the MOE curriculum for primary grades 1-6, but also to make it pertinent to out of school youth. As two years of the MOE curriculum was condensed into one year (two semesters) of the AYP curriculum, AYP went through several stages to develop, revise and finalize it. The field data demonstrated that the AYP ABE curriculum has been well received by the learners, facilitators, administrators; and the MOE officials who have interacted with it. The facilitators and administrators were in agreement that once the curriculum was revised (for Level 1 and 2) it was more pertinent for the learners. The response to the Level 3 curriculum by these target audiences was also positive, and no respondent stated that it needed to be revised. Many administrators and facilitators (along with AYP staff) also pointed out that a major positive aspect of the curriculum was that it contains teaching methods that are student centric and involves group activities to keep the learners engaged, rather than being a curriculum that requires the teacher to lecture and the students to take notes, as is the case in the MOE day schools. The field data also revealed certain patterns with the use of the curriculum in the AYP schools including:  The learners who had never been to primary school (hence have not had any prior education) had more challenges with the curriculum when they started Level 1.  The facilitators and administrators interviewed for this evaluation felt that the Level 1 and 2 learners had the most challenges with the numeracy lessons.  There was a lot of material to cover in each class along with the fact that learners had questions, clarifications were required, some learners were slow, others came late so the facilitators had to repeat certain topics etc. Therefore, many facilitators were of the opinion that it was difficult to cover all the subjects for an allotted class in 3 ½ hours. This led to facilitators having to cover materials that had not been covered in one class Photo 4: AYP Learners in Montserrado County 32 in the next one, leading to delays in covering the curriculum for each level.  Facilitators had to be especially sensitive in discussing some topics in the curriculum (reproduction, sexual health etc.). The IAI component of the AYP curriculum was also well received. Many facilitators and administrators said that it helped the learners in hearing correct English, while also giving them an opportunity to improve their pronunciation and to discern sounds. Other facilitators noted that it helped having the IAI lessons as they broke the monotony of learning from the textbook. The IAI lessons were also broadcast over radio during the Ebola crisis; however, no learners covered in the evaluation stated that they had listened to these broadcasts. The field data also revealed that the learners and facilitators were less engaged in the mLearning component of the curriculum. Further in one AYP school, the administrator also said that mLearning did not take off as few learners at the site had cell phones. Facilitators and Administrators The consensus among the facilitators and administrators was that the trainings they received from AYP were good. Initially they received a ten-day training and then were offered follow up cluster level trainings based on their needs as observed (or stated to) by the AYP county coordinator; who visited each AYP school in his/her county twice a month.3 The facilitators and administrators were also of the view that the AYP trainings were adequate to prepare them to teach the AYP ABE curriculum, and if they had any major queries, they could pose them to the county AYP coordinator. When asked what main benefits they received from the trainings, they provided the following feedback:  Capacity to teach ABE  Getting more expertise to teach their subjects in their levels  Learning student centric methods of teaching and how to better engage with learners  Including group activities in teaching the curriculum  Improving their teaching methods and skills  Getting their questions addressed and queries clarified Some facilitators also pointed out that learning student centric teaching methods for the ABE classes had helped them improve their teaching in their MOE day school. 3 Some facilitators got more follow up trainings than others including through Stella Maris Polytechnic University. “The whole class takes 3 1/2 hours, but in that time I cannot cover all the literacy and life-skills lessons I am supposed to. Some learners are slow, and when they do exercises it takes a lot of time to complete them. This is my main challenge with the curriculum.” - Male Facilitator, Level 1, Montserrado County “The AYP trainings increased my knowledge of teaching, how to talk to learners and how to present a lesson. I improved my teaching skills.” - Female Facilitator, Level 2, Grand Bassa County 33 The facilitators and administrators were also in agreement that the revised Level 1 and 2 curriculum was improved and more useful for the learners. Further, they did not have any issues with the facilitators’ guidebooks. Nonetheless, certain issues did emerge over the course of the evaluation including:  The capacity and experience of the facilitators and administrators is not uniform, and even the provision of extended trainings cannot change this fact. Thus, some facilitators were better prepared and had more capacity to teach the AYP curriculum than others.  The facilitators were used to teaching with methods whereby they lectured and students took notes, so shifting to student centric methods was challenging for some of them.  Some facilitators were of the view that having slow learners in their classes impacted the overall quality of the learning for all the learners, as the slow learners’ required individualized attention.  The administrators were supposed to oversee the facilitators and to aid them in improving their teaching. However, in many cases the administrators just made sure the facilitators were coming regularly and teaching, without providing them any feedback on how they were teaching.  The vast majority of the facilitators and administrators in the AYP schools are male, which is a reflection of the MOE teachers in the day schools and not a result of any hiring bias by AYP. In the FGDs with learners, no complaints were voiced against the facilitators or administrators. In fact, the learners were complimentary of the facilitators and said that they were helpful, clarified issues and made sure learners understood the topics in class. The learners also said that in the beginning when they started a level, certain topics were hard to learn, but the facilitators made learning easier by explaining the topics in different ways. Completing AYP ABE Levels Data was collected from learners, facilitators, administrators and AYP staff to understand what factors enabled a learner to successfully complete the level he/she was enrolled in. Successful completion of a level meant passing the completion test administered at the end of the academic year for each level. The evaluation findings demonstrate that the following factors had an impact on a positive outcome of level completion:  Personal commitment  Regular class attendance  Active engagement with the curriculum and in class activities (including in group activities), and expressing yourself in class  Asking questions in class and getting clarifications  Focusing on and completing the workbook exercises  Having a purpose to attend the classes (enrolling to learn to write your own name, to get some short-term skills or long-term livelihoods trainings etc.)  Active participation in youth clubs and the ability to generate income from youth club activities (making soap and selling it or selling vegetables grown in the youth club garden)  Having support from families and peer support from other learners  Having capable facilitators who take time to explain and re-explain concepts that learners find hard to understand, who engage the learners and who answer their questions  Having engaged ABE committees who support the school and motivate the learners “The facilitators are good and very helpful. Whenever we have any problems or questions, they take time to explain so we understand.” - Level 1 and 2 Learners, Grand Bassa County 34 The feedback from these sources also emphasized that there was no gender difference between learners who successfully completed a level. Overall, however, the one dominant factor (apart from personal motivation) that contributed to a positive outcome for the completion of an ABE level was the learners engaging in (or to have the potential to engage in) short-term skills or long-term livelihoods training. Learners who benefitted from short-term skills trainings (for example soap making) and who were making and selling this soap through active youth clubs were more likely to compete a level; as were a smaller subset of learners who had long-term livelihoods trainings (at the Liberia Opportunities Industrialization Center - LOIC), as one requirement for them to complete their training was to regularly attend the ABE classes, which would ultimately lead to the completion of that level. Out of School Youth Improving Their Learning Learners had different motivations to enroll in the AYP classes including the following:  To be able to write their own name  To be able to do some numeracy for work or to start a small business  To improve their existing literacy or numeracy  To be able to enroll in a formal MOE school in grade 7 (after completing the 3 levels)  Get some short term skills or long term livelihoods trainings to earn an income Taking into account the different motivations of the learners to enroll in the ABE classes, their learning outcomes could be determined from different sources including:  The formal OLA and EGMA assessments  Completion tests at the end of each level  Assessments at the end of each module in the curriculum administered by the facilitators to the learners  The workbook exercises for each lesson that learners complete Even though the OLA and EGMA assessments done in 2012/2013 showed limited improvements; the facilitators, administrators and AYP staff (and the MOE officials who visited the AYP schools) were clear in stating that if learners were regular in class, they made significant gains in their learning from the start to the end of a level. They further based this feedback not only on the completion tests, assessments at the end of modules and learners work book exercises; but on their own personal experiences of observing the qualitative improvement in the learner’s literacy, numeracy, life skills and work skills, as demonstrated in the classroom over the course of the time taken to complete a level. The learners themselves in the FGDs were very vocal and clear that AYP had helped them to learn and to improve their literacy and numeracy skills, and they were very thankful for the ABE classes. Many learners shared personal experiences of their improvement in literacy and numeracy and how that has had an effect on their lives. The act of being able to write their name was an empowering experience that gave them self-confidence and pride and which transformed their life in multiple ways (they could “The AYP has definitely helped the learners. I see a big change in the learners from the time they join a level to the time they end the level. For level 1, when learners joined they can’t even write their name, but by the end of the level they can write, recognize the alphabet and put numbers together. For level 2, when learners join they can’t compose sentences, but by the end they can.” - Administrator, AYP School, Montserrado County 35 see their names on lists, sign in on sheets when going to offices/clinics, sign checks, avoid being cheated when filling forms, sign certificates etc.).4 Apart from literacy and numeracy (which are the dominant subjects), the learners, facilitators and administrators also appreciated the life skills and work readiness courses. In the FGDs, the learners stated that they had benefitted from the work readiness subjects by understanding how to assess job opportunities, differences between types of employment etc. The facilitators teaching this subject noted that it helped learners to understand leadership, communication and team work – skills which would help them in their jobs and lives. The learners, facilitators and administrators were also complimentary of the life skills subjects as these affected the day to day lives of youth. Many learners also commented that they had learnt about health related issues, reproduction, diseases and prevention measures, community services etc. from these lessons. It should also be noted that learners were exposed to lessons on healthy behavior and living not only through the life skills curriculum, but also through visits from representatives of PSI-Healthy Actions, an AYP partner. Due to these inputs, learners got a better education about sexually transmitted diseases, family planning, etc. Two AYP schools (one rural, the other urban) covered in the field visits were also offering Grow Your Business (GYB) in Level 3. Two of the four days of the GYB classes were devoted to agriculture and two to business. One of the two days for the agriculture classes was theoretical while the other was practical, when the facilitator and learners went to the youth club garden and implemented what they had learnt. Discussions with the GYB learners and a facilitator in the rural school demonstrated that this course was beneficial, and the learners were especially keen to gain information and skills to do some type of agriculture related business in the future. However, in the urban school, the response to GYB was more tepid, and the feedback obtained was that learners were less interested in agriculture. It was difficult to get learners to come to the garden to implement what they had learnt, and work done by the learners at the garden was ruined by thieves in the night. 4 Many of the Level 1 learners also said that they found the IAI program to be helpful in improving their English. “I did not even know ABC before I joined AYP. AYP has opened my eyes after I learnt how to read and write. I feel like I have come from being a caterpillar to a butterfly.” - Male Learner, Level 2, Grand Bassa County “I like AYP because it gave me a chance to study. It allowed me to show my dedication to school and to do something for myself. AYP is making me somebody...” - Female Learner, Level 2, Grand Bassa County “The radio programs help me to understand English better and also to speak English words better.” - Female Learner, Level 1, Grand Bassa County 36 Question 3: Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training? The AYP model of providing livelihood trainings is effective, and is considered the paramount activity that encourages learners to participate in the program and incentivizes their continued participation. In support of livelihood skills development opportunities for learners, AYP created partnerships with institutions that are involved in skills training activities in Liberia. These included long-term skills training in partnership with the Grand Bassa Community College (GBCC) through the United Nations Industrial Development Organization (UNIDO), and the Liberia Opportunities and Industrialization Centers (LOIC) in two locations: Grand Bassa and Montserrado Counties. Short-term skills training was facilitated by the Centers for Skills Innovation (CSI), implemented through World ORT in three counties (Lofa, Nimba and Bong). Initially, some learners expected that AYP would pay stipends for their participation and expected to be financially compensated for the time they invested in learning. In response to this issue, AYP introduced the livelihoods intervention in order to provide knowledge and skills for learners that would enable them to generate money post-training activities. From a practical standpoint, this intervention was the key method of developing youth livelihood skills and supporting enhanced access to employment opportunities. Although unemployment is high among the youth population (among youth 25-34 years in age, unemployment is 69.7 percent in urban areas and 72.4 percent in rural areas5), many youth see one￾off training programs as an opportunity to generate money and do not place as much value on the academic knowledge they would acquire through the literacy and numeracy curriculum. However, in most cases, some basic literacy and numeracy skills are required in order to engage in successful livelihoods (for example, basic math skills to make change and keep records of transactions for marketers). The livelihood skills training made AYP a distinct program among all other adult literacy or out of school programs that have been implemented in Liberia. It served as a main pillar in motivating learners to develop interest in the program and to complete their levels successfully. Learners come with high hopes of acquiring skill trainings, coupled with the literacy and numeracy skills they would gain. Considering their ages, most of them wished to obtain marketable skills that will enable them to provide 5 Liberian Institute of Statistics and Geo-Information Services. Report on the Liberia Labour Force Survey 2010. February, 2011. “In the GYB class I have learnt how to separate business money from family money, how to keep records, make receipts… all these things we did not know before, and they will help us to do our business better.” - Male Learner, Level 3, Grand Bassa County “There have been a whole mass of ABE programs in the past, and there is a huge demand on the part of the learners, but the key thing is to link it to livelihoods. People need this; they don’t have time and resources to learn for learning’s sake.” - Former USAID Staff associated with AYP 37 daily meals for their respective families. Some had their expectations met while some others were not: those who were primarily interested in gaining skills training were surprised to realize that the AYP lessons were more academic than they expected. Short-term Skills Training In order to ensure that training opportunities met both the needs of trainees and demand in the labor market, a youth-led Labor Market Assessment (LMA) was conducted to identify in-demand skills and growth sectors. This was done in collaboration with the Ministry of Education (MoE) and local organizations to increase access to social opportunities, leadership development, and livelihood training for participants. Based on this LMA, participants received training in soap making, bee-keeping, food preservation, candle making, snail raising, pastry-making, etc through the short-term skills training. Two or three participants per community went to central training sites in nearby localities for a two-week training and were known as Peer Training Innovators (PTI). These PTIs then returned to their home communities to pass along the knowledge and skills to their peers. As a result, nearly all AYP learners who were actively engaged in the youth club and attended classes regularly were also afforded short-term livelihood opportunities in their communities. According to data reported in PIDS, a total of 3,261 learners (1,005 males and 2,256 females) benefited from the short-term skills training programs since AYP’s inception. As mentioned above, skills training was used as retention strategy which yielded fruitful results in maintaining learners in their various classes until completion of their levels. Based on information reported in PIDS, the project has had 3,103 learners complete the level for which they enrolled (14 percent of the 22,256 total participants enrolled in AYP). In addition, AYP graduated 197 learners from Level 3 last year, which was the first year that the project had Level 3 completion testing; additional graduates were expected at the time of the evaluation’s data collection, which took place a few weeks before end-of-year completion tests were conducted. AYP staff reported that they conducted exit interviews on a limited scale, asking learners about their plans for next steps; most of them said they planned to pursue either technical and vocational education and training (TVET) or formal education as their next option. Long-term Skills Training Learners at the LOIC received long-term skills training in areas such as tailoring, carpentry, auto￾mechanics, masonry, small engine repair, electrical, and plumbing, etc. Learners at the LOIC trained for 16 months to complete the vocational training program: the first 12 months comprised of class work at LOIC, learning both theory and practice. Upon completion, they were placed into an internship by LOIC at various entities based on specialty, lasting four months. Internship opportunities for long-term skills training (such as LOIC) were limited in scope, with a total of 242 LOIC trainees overall, which represents about 1% of all AYP learners enrolled. Overall, learners’ attendance remained high while simultaneously attending ABE and technical classes at LOIC, although there were some reports of individuals dropping out of classes after starting the LOIC training. Based on lessons learned from the first phase of long-term skills training (which offered no transportation stipends) AYP provided transportation stipends to learners in the second round of training to support their attendance at both LOIC sites. In general, these stipends supported learners brought from remote villages and towns by AYP to enroll in LOIC skills training programs, who were then more successfully retained and had more regular attendance in classes. 38 However, there were some instances where beneficiaries reported inconsistent receipt of transportation stipends. These reports may have been linked to the first round of training, in which there was no memorandum of understanding to tie the project in providing transportation reimbursement for its learners. This information was confirmed by AYP that there was no point in time did the activity promise to provide learners with accommodation prior to their relocation to a new community/town or city, and that learners signed a “LOIC/Advancing Youth Learners Interview Form” that clearly stated that the learner will be held responsible to identify someone or a relative in the city that will be able to accommodate him/her during the period of stay for the sole purposes of enrollment at LOIC. LOIC hosted graduation events for the learners, and AYP initiated the aforementioned tracer study on LOIC graduates to determine whether they continue with work and/or gain employment in the trade area they were trained. At the time of this evaluation, results were not yet available. Besides LOIC, AYP had a small set of learners, about 25, through GBCC in 2012 -2013. With capacity support from UNIDO, this long term skills training focused on heavy duty mechanics. Those that enrolled successfully graduated from the training. The partnership discontinued because UNIDO was in Grand Bassa for a one year (2012 – 2013) period. Photo 5: Young woman studying small engine repair at the LOIC in Grand Bassa county In the case of both the short- and long-term skills training, qualitative evidence indicates that AYP did an excellent job of aligning training to the needs of women. This was evident in many forms throughout the field work: the percentage of female participants who were enrolled in the program overall suggests that their needs were being met and word-of-mouth was encouraging increasing participation; the number and percentage of women engaged in short-term skills training and acting as PTIs indicates that these skills were highly relevant to their needs and interests; and the number of women who were engaged in less-traditional skills, such a small engine repair (pictured above) with enthusiasm and pride for their capacity to undertake new fields. The evaluation team also heard anecdotal evidence of all- 39 female car repair shops being opened by graduates, and other stories of females having great success in the livelihoods component. It is expected that further evidence of this success would be available in the quantitative data gathered through AYP’s forthcoming tracer study. Internships AYP also introduced an in-kind internship incentive program in five counties as a pilot project in January 2016. This provided selected learners with hands-on work experience and skills in market-driven industries. AYP conducted a market survey to identify service providers who would be willing to sponsor learners as interns. A total of $100 USD of in-kind support per learner was given to each sponsor in tools and skills, which incentivized sponsors to participate and provided them an opportunity to grow their own business capacity as well. This three-month internship program was designed to complement the large need in long term skills training desired by most learners. A small subset of 75 AYP learners benefitted from the opportunity across the five counties, and another round of internship placement was ongoing at the time of the evaluation, allowing more AYP learners to benefit. This was expected to benefit an additional 30 learners from each of the five major counties before the end of the project. Youth Club Gardens and Grow Your Business Agriculture, forestry, and fishing is the most prevalent occupation in the Liberian labor market, with nearly half (47.3 percent) of all workers engaged in this sector. 6 Thus, including agricultural training in AYP was an essential element of the livelihood skills training. Nearly all of the 147 AYP sites have a garden that is managed by the youth clubs, which provide some income for the organization. In an expansion of this work, AYP developed the GYB activity and implemented it in 18 sites, which included a curriculum and training for learners to expand their agricultural knowledge beyond traditional subsistence level to commercial-based farming, including marketing and financial management of agricultural endeavors. Learners benefiting from the GYB curriculum reported that their knowledge expanded, and their ability to implement farming beyond subsistence level increased. Although they gained the knowledge and have the land for larger-scale farming, they expressed a need for tools and seeds to implement this knowledge, which is difficult for AYP to provide without a waiver. 6 Liberian Institute of Statistics and Geo-Information Services. Report on the Liberia Labour Force Survey 2010. February, 2011. AYP did a lot of good things for us especially me. I was in the interior, knowing nothing about mechanic work. AYP brought me in the city of Buchanan and allowed me [to enroll in an AYP school] at the same time engage with the LOIC classes where I graduated in auto-mechanic/light duty. I can now open car engine and do some repair work. Additionally, I was placed in an internship at the RB Company, where I am yet attached as a Mechanic after my internship period. - Male AYP Graduate, Grand Bassa County 40 Ministerial and Cross-sectoral Support At the current time, there was no evidence of inter-ministerial cooperation to support youth livelihoods development. The generally understood approach is that the MOE is responsible for the formal education system (including vocational training) and the MOYS manages the informal system, and there is no sharing of materials, teacher training, curriculum, or other assets between them. This silo-ing of services is, to some degree, echoed in the donor market, both intra-agency and between donors, although both systems are improving, as evidenced by the multi-sector funding provided for AYP, and in the adoption of AYP materials and implementation strategies by the EU for their forthcoming ABE program.  To what extent did livelihood trainings lead to quality employment for youth? Placements for low-skilled youth are possible, as 74.4 percent of businesses surveyed in the Labor Market Assessment (LMA) said they would be interested in hiring on-the-job trainees. The nature of these businesses and informal private sector will shape the mechanisms and processes in which youth identify and receive on-the-job training, internships, and full-time job placements. A total of 136 graduates were reported in PIDS by AYP as employed, although this data is from Year 3 and outdated. Most AYP graduates could not be traced to know their current employment status. However, most reported being self-employed rather than attaining formal employment. On the other hand, there are individual-level cases of learners whose participation in AYP supported their transition to gainful employment. Without more recent outcome-level data from AYP beyond the Year 3 figure of 136 youth employed in PIDS, the evaluation was not able to determine how many participants gained employment as a result of AYP training. Additionally, learners were provided with internships and linked with key service providers, but outcome level data on how frequently this led to gainful employment was also not available. Based on qualitative findings, these experiences were limited in their ability to provide strong pathways to consistent employment or entrepreneurial endeavors. Opportunities for youth livelihoods are challenging especially with limited formal employment sector opportunities in Liberia. AYP was pleased with the services provided by LOIC especially in career guidance training. This successfully led to the reduction of oversaturated markets, as career guidance can steer learners toward more in-demand skills areas. With the high number of cases of skills training courses, the project could not provide the necessary materials to support transition to employment in these areas. AYP is a good program. I was a fisherman when I first came to know them. AYP classes prepared me in simple mathematics, business and life skills principles. Upon completion of level 3, I joined the UNIDO program through AYP connection … I did light and heavy duty mechanics, a course that lasted for 1 year, eight months. I was certificated upon completion of the program and served the UN system as a contractor for three consecutive months. Today, I am self-employed and have my own garage with nine men working under me as apprentices. Male AYP Level 3 and UNIDO Graduate 41  To what extent did learners improve their literacy, numeracy, and work readiness skills to date? This section examines the literacy and numeracy assessment tools employed by AYP, the factors contributing to positive outcomes for learners to complete an ABE level, and the improvements in learning of the targeted youth. AYP EGMA, OLA and Work Readiness Assessments AYP employed the EGMA and OLA assessment tools to measures learners’ improvements in their numeracy and literacy skills. These tools were initially employed in 2012/2013 and will be administered again before the project ends (2015/2016). A work readiness assessment was also carried out in 2012/2013 as part of a larger livelihoods survey. Early Grades Mathematics Assessment (EGMA) 2012/2013 EGMA is a tool used in many countries to measure foundational numeracy skills. It is administered orally and assesses learners’ in various subtest areas including: number sense including number identification; CASE STUDY: REUBEN JUNIUS, AYP GRADUATE When Reuben was 11 years old, he left school. But in 2012, he joined AYP at SOS School #2 near his village. After completing Level 2, he moved to Buchanan and completed Level 3 at Benson River School, so that during his final year of study, he could enroll at the LOIC to train as a small-engine mechanic. During his final year of AYP, Reuben worked very hard. He attended theoretical studies at LOIC from 8:00 a.m. to 3:00 p.m. every day. After the LOIC training, Reuben interned from 3:00 to 5:00 at the 106 Garage to develop his practical skills, and then attended AYP literacy and numeracy classes from 5:00 until 8:00 p.m. On Saturdays, he worked all day at the garage to expand his knowledge. After graduating from AYP, he was hired as an employee at the 106 Garage. Reuben says that AYP was a good experience for him: before he joined the activity, he did not know how to write his name, but AYP taught him the livelihood skills that he needed to find a job. He sometimes returns to the LOIC to talk to the AYP students about his experience. He says that some of the learners from the LOIC have returned to their villages in the bush as they could not afford to live and work in Buchanan for extended periods of time. In the future, Reuben plans to go back to the LOIC, to get additional training in electric systems, so he can combine it with his mechanic experience. This will allow him to do more and different work in the garage. 42 discrimination, patterns and knowledge of fundamental mathematical operations such as addition, subtraction, multiplication and division; and ability to identify basic shapes. The AYP EGMA was designed to measure results in improving numeracy skills of learners and aimed to answer the following questions: Do youth participants demonstrate improved math skills at the end of ABE Level 1? Were there any differences between men and women? Detailed subtest results are available in Annex VII: OLA and EGMA Subtest Results. The evaluation design was quasi-experimental including a pre- and post-test design with one intervention group. The tested sample was randomly drawn from participants of one unique AYP Level 1 cohort who started their ABE classes in September/October 2012. The same participants were assessed at the endline stage in May/June 2013. Level 1 learners were chosen from 18 AYP schools; Table 6 enumerates them by county. Table 6. Number of Learners Assessed with EGMA by County Bong Lofa Montserrado Nimba TOTAL Learners Baseline 63 60 69 70 262 Endline 42 46 38 50 176 Consistent with the proportion of youth enrolled in AYP, test-takers were predominantly female (78 percent) and rural (60.8 percent); and the majority of youth, both male and female, were between the ages of 26 to 35 (66.5 percent). The majority (68.2 percent) of respondents had also not attended school before. Of those that had gone to school, the majority (97.1 percent) did not progress beyond grade four. The data analysis of the EGMA results showed that:  Level 1 youth who participated in AYP ABE classes made statistically significant gains in their numeracy skills (at p <.001 level) between the pre-test and the post-test.  The test of significance of change between pre-test and post-test revealed that the Level 1 youth showed statistically significant improvement for all subtests except one – division – on which the scores remained relatively flat.  The largest gains were observed in simple one-digit additions (11 percent improvement) and one-digit subtractions (10 percent improvement).  Males significantly outperformed females in all subtests except shape recognition (at p <.001 level). Males overall gained more during the course of the intervention on the majority of subtests.  Statistical test of significance shows that the improvements were much more pronounced among participants from rural area. Among participants from urban areas changes on half of the subtests were not statistically significant and could be due to a chance variation in the sample.  The percentage of youth who were not able to answer any questions correctly was particularly high on the following subtests: missing number, double-digit subtraction and addition, and multiplication and division. However, even with statistically significant gains, the overall scores of the tested youth were still very low when combined for all the subsets of the EGMA tool. The mean scores also demonstrated low achievement across individual subtests, indicating very low numeracy skills. Therefore, while the EGMA results showed that youth gained in all areas of the test, the gains were relatively small, and overall scores were low. Figure 2 below illustrates their results by subtest area. 43 Figure 2. EGMA Subtests Mean Gain Scores of Learners from Baseline to Endline (in Percentage) *Division: the gain was less than 1 percent, and was not statistically significant. Out of School Literacy Assessment (OLA) 2012/2013 OLA is designed to orally assess the reading skills of older youth and young adults in developing countries. OLA assesses subtests including some aspects of phonics, word recognition (decoding), oral reading fluency, comprehension and some real-world literacy skills. The OLA assessment was done to test whether AYP ABE classes were successful in effecting a positive change in Level 1 learners’ performance in targeted literacy skills. The evaluation question stated: Do youth respondents demonstrate improved literacy skills at the end of ABE Level 1? Were there any differences between men and women? Detailed subtest results are available in Annex VII: OLA and EGMA Subtest Results. The design employed for this evaluation was quasi-experimental using pre- and post-tests with two intervention groups and a comparison group including:  AYP Level 1 group who did not receive IAI instruction  AYP Level 1 group who received IAI instruction in additional to the regular curriculum  Comparison group (receiving ABE from NAEAL) Table 7: Number of Learners Assessed with OLA AYP non IAI Learners (Group 1) AYP IAI Learners (Group 2) Comparison Group (NAEAL) (Group 3) TOTAL Learners Baseline 567 198 181 946 Endline 422 138 151 711 The baseline test was administered to these three groups in October/November 2012. The same three groups of youth were tracked over the course of the ABE year and were tested 8 to 9 months after the baseline between May and June 2013. However, in the case of the control group, the total number of 5.9 5.9 2.8 11 7.1 10.1 7.1 3.3 6.06 0 2 4 6 8 10 12 Number Identification Number Discrimination Missing Number Addition Part 1 Addition Part 2 Subtraction Part 1 Subtraction Part 2 Multiplication Division* Shape Recognition 44 months in class varied because each start date depended on the site. The comparison group’s range of exposure was one to three months. More than three-quarters of the intervention group respondents were female, a figure that was consistent with overall enrollment trends, and the comparison group was largely female (90.6 percent). The median age of the intervention group was 29 and 37 for the comparison group. A little more than half of the total respondents (57 percent) were rural and 43 percent were peri-urban and urban. Over three quarters (77.9 percent) of the comparison group had never attended school before, whereas 60 percent of the combined intervention group respondents had never attended formal school. The majority of respondents reported that they spoke Kpelleh at home (51.4 percent of all groups), while less than 10 percent spoke English at home. The data analysis of the OLA results showed that:  Group #2 (AYP learners with IAI) had a significant improvement in their overall scores, moving from 17.5 percent correct across all OLA subtests at baseline to 23 percent correct at the endline.  However, there was an overall low achievement on the majority of OLA subtests among all the Level 1 participants (both groups), who scored on average only 20 percent correct across nine subtests. This result remained virtually unchanged between the baseline and the endline.  Learners from the comparison group showed no change in scores between the baseline and the endline.  All learner groups showed some improvement on letter naming, but only 53 percent could correctly identify more than 19 letters, a skill that is a pre-requisite to reading.  There were very low scores on letter sounds, with less than one percent of learners able to sound out all or almost all letters.  Urban AYP learners (who had the IAI intervention) showed the most improvement in letter sounds with a six percentage point gain.  Of all the AYP Level 1 learners, urban learners also showed the most improvement in reading words in a connected text, more than doubling their results at endline compared to baseline.  Young men performed significantly better than young women, at the p < .001 level. Both young men and young women in group #2 (IAI) improved significantly between the baseline and the endline. However, young women attending group #1 (non-IAI) did not improve at all, and young men in group #1 showed only slight improvement. The table in Annex VIII: OLA and EGMA Baseline Study 2015/16 provides additional detail on the above findings. Figure 3 below illustrates learner results by subtest area. 45 Figure 3: OLA Subtests Mean Gain Scores of Learners from Baseline to Endline (in Percentage) After the EGMA and OLA endline assessments were administered, AYP implemented a series of decisions to further improve the literacy and numeracy skills of learners. These included:  The Levels 1 and 2 curricula were revised to be more appropriate for beginning English language learner, and for learners with very low numeracy skills.  In facilitator trainings, more emphasis was placed on teaching and learning activities in the foundational reading skills associated with decoding and to enhance instruction and practice in reading comprehension; and in the foundational skills of number discrimination and pattern recognition (missing number) and to enhance instruction in simple basic operations.  IAI instruction was developed and provided for all subjects in Level 1 (instead of just literacy) to help facilitators teach difficult concepts.  Facilitators and administrators were trained to slow the pace of teaching literacy and numeracy and to spend more time on these two subjects in Levels 1 and 2.  Closer monitoring of facilitators and learners was carried out to ensure that facilitators were carrying out their instruction as per the curriculum and the instructional strategy and that learners were being exposed to the entire curriculum. OLA and EGMA baseline and endline tests are scheduled for the 2015/16 academic year and were in the process of being completed at the time of this report. The baseline study results are available in Annex VIII: OLA and EGMA Baseline Study 2015/16. Work Readiness Assessment (2012/2013) 7 In order to better understand the livelihoods situation of youth living in vulnerable situations, AYP conducted a livelihoods survey in 2012/2013. A scenario-based work readiness assessment was also 7 Work readiness skills are loosely defined as the collaboration, critical thinking, decision-making, and interpersonal communications skills and behaviors that help people succeed in a range of work settings. -10 -5 0 5 10 15 20 AYP non IAI Learners (Group 1) AYP IAI Learners (Group 2) Comparison Group (Group 3) 46 administered (as part of this survey) to measure changes in participants’ work readiness skills. The work readiness assessment is a situational judgment or scenario- based assessment of selected work readiness skills. Test-takers are given a short scenario, accompanied by illustrations, about which they answer multiple-choice or yes/no questions. The evaluation design was quasi-experimental with a pre- and post-test and one intervention group and one comparison group. The intervention group consisted of participants from the AYP ABE Level 1 cohort who started in September/October 2012. There were nine months between the administration of the pre and posttest surveys. The comparison group had relatively similar socio-economic characteristics to the intervention group, with the exception of gender. The comparison group was 98 percent female while the AYP ABE learners were 79 percent female. The comparison group learners were recruited from a youth/adult literacy program operating in two of the same counties as the ABE intervention. Table 8 enumerates the learners completing the Work Readiness Assessment by county. Table 8: Number of Learners Completing Work Readiness Assessment by County Bong Montserrado TOTAL Learners AYP Learners 115 128 243 Comparison Group 81 97 178 The results of the work readiness assessment showed that:  Youth who participated in AYP ABE classes scored higher at post-test than the comparison group, despite starting slightly lower than the comparison group at baseline.  The mean gain score for the AYP group was over three times as large as the comparison group.  Participation in ABE is associated with increased competencies in work readiness compared to those who do not take ABE. However, reliability tests done at endline indicated a low internal consistency of the assessment tool overall. This may have indicated that the testing format was not appropriate for learners with very low literacy levels. Question 4: Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions (based on what has been learned from the accomplishments and challenges of AYP up until now) that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? In responding to this evaluation question, it is essential to reiterate that there are gaps in the overall information available to the evaluation team, in order to identify the most effective AYP interventions upon which to build. There was no mid-term evaluation to assess the alignment of AYP’s work to the communities’ and Ministry’s needs (USAID and AYP made a conscious decision to not conduct a midterm evaluation based on strategic planning advice), and very few project-level outcomes have been reported in the PIDS system half-way through Year 5. As AYP concludes its implementation lifecycle, additional information on learning outcomes achieved by the project, as well as the project’s success in supporting effective linkages to relevant and realistic livelihoods opportunities should be examined. In addition, results from the institutional and organizational capacity building would enable a more informed assessment of how future interventions could work more effectively with governmental and non-governmental entities to ensure quality service provision and scalability. 47 Literacy and numeracy curriculum: Based on the information available for this performance evaluation, early indications suggest that the literacy and numeracy curriculum provided by AYP are effective learning tools to address learners’ needs. The 2012/13 OLA and EGMA studies demonstrated limited knowledge gains, but this was before the curriculum revisions, which according to learners and facilitators interviewed, was a significant improvement in the learning experience, especially for Level 1. Based on the qualitative feedback, the evaluation team felt that the AYP classes were helping learners to improve their literacy and numeracy skills. The forthcoming endline studies for these tests will be extremely important to illuminate how effective the classes are. Incorporation of skill development: Based on qualitative feedback, one of the most effective interventions for the recruitment and retention of learners, and development of their livelihood skills, were the short-term skills training, and for a smaller sample of learners, the long-term vocational skills training programs. Based on inputs from a wide variety of respondents, the program worked to address the issues of recruitment, and most especially retention, with the best strategies available to them. Nonetheless, external challenges limited their overall efficacy, especially in regards to the potential to offer long-term skills training for a larger percentage of the participants. The inclusion of Economic Growth funding did enable the project to offer vocational skills training for 242 learners, and the individuals interviewed for this evaluation who participated in this training all reported that the experience was very valuable in building their capacity to access relevant and realistic livelihoods opportunities. Women and rural communities: Another element that was clearly effective, based on available information, was the ability of AYP to reach female participants and to meet the needs of learners in rural communities. Based on the data reported to PIDS, the project reached a very high percentage of women: the aggregate proportion of 58 percent females enrolled is slightly misleading, as the first year of implementation, only 25 percent females were enrolled; since that time, the project has averaged closer to 70 percent female enrollment rate. It is important to note, however, that female participants do require gender-sensitive retention approaches, as detailed earlier in the evaluation report. While PIDS does not offer disaggregated enrollment figures based on urban or rural settings, the sense of engagement and enthusiasm that was observed among the most isolated rural communities visited by the evaluation team was a marked contrast to the attitudes of learners at some of the urban sites. Based on participant feedback, retaining learners in rural communities requires special attention to accessibility of training venues and also the alignment of skills training to the local contexts (e.g., Grow Your Business farming training is more relevant to rural contexts, and bee-keeping training should be implemented in communities where bees are prevalent). Working with the MOE: The strategy of working through the MOE to implement ABE activities, rather than going solely through local NGO service providers, does pose a greater challenge, and the experience of AYP in promoting ABE as a priority area in the MOE’s strategic plan demonstrates how difficult it can be to push forward a policy agenda that lacks political will. Nonetheless, they are the paramount partner in all educational service provision, and the inclusion of and alignment to MOE is crucial in building sustainability and scalability into project implementation strategies. We should not leave the MOE behind, as they are tasked with conducting the educational service provision in this country. They are under-resourced and have limited capacity in many respects. And they can also serve as a barrier to moving forward in our activities if they are not fully involved, so it’s very important as we go to find ways to drag them along with us. We have come to realize when they become dispassionate about things, they have clever ways to make it difficult for this process to move forward. - AYP Staff Member 48 CONCLUSIONS Question 1: Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? Based on the available evidence, the evaluation team concluded that while AYP was successful in building capacity of individuals within the Ministry to plan, deliver, and measure ABE for youth, the institutional￾level impact of these efforts were limited by the challenges of limited political will and high turnover among Ministry staff. The curriculum and supplementary materials have been provided to the MOE, but their planned institutionalization within the standard academic system was not implemented to scale. Ministry staff accompanied AYP on joint monitoring visits to learn each other’s systems and techniques for monitoring ABE classrooms, but the Ministry’s tools were not upgraded and MOE reporting systems remain the same. In addition, AYP enhanced the MOE’s capacity to train teachers and administrators to offer high quality ABE training through the development of nearly 300 MOE school administrators and almost 1,000 MOE teachers trained to date. The establishment of the teacher training program for ABE facilitators at Stella Maris Polytechnic offers an exciting opportunity for the MOE to expand the provision of adult education to as many learners as possible. Finally, AYP’s work through NGO service providers like NAEAL and LIYONET explored different models of service provision and enhanced the organizational capacity of these organizations to expand their own work in ABE. This represents an opportunity not only for the MOE, but for other donors as well, to scale up ABE learning to more closely match the very strong demand among learners for these types of opportunities. The MOYS also represents a strong potential partner that is eager to engage with donors and to expand their capacity to work with out-of-school youth; there is a need to establish better pathways to expand their integration into ABE work, especially in livelihood skills training. o To what extent were the Government’s expectations and policy needs met with AYP? Overall, the MOE indicated strong satisfaction with AYP as it was structured, and were satisfied with its delivery. Some respondents from the Ministry would have appreciated more involvement in the design and planning for AYP and should be consulted as much as possible in all stages of project development, in order to increase their buy-in and uptake of projects. However, this is likely due to high turnover in the Ministry impacting the institutional memory, as other stakeholders testified that Ministry officials were highly involved in this activity. Nonetheless, this underscores the essential need to ensure that the MOE feels ownership of activities and their implementation. o How likely will the Government be to use the ABE materials in the future? The MOE has not provided accreditation for ABE classes to date; nor is there any specific indication that they plan to use the AYP assets, either immediately or in the long term, on a wider scale than is already being implemented. Nor is there an indication that the MOE is working to certify ABE learners within their own programs or to provide this for AYP learners on an institutionalized basis. The Ministry feels that their first priority must be quality within the formal education system and that they lack the 49 resources do to this properly, let alone respond to the very high demand among out-of-school youth for second-chance educational opportunities. There is a strong desire among youth who lack basic literacy and numeracy skills for ABE opportunities in general and for certification of their achievements when they do have these opportunities. Some forward-looking CEOs and DEOs are making this happen on a localized basis, and there is potential for wider implementation, which respondents indicated can be encouraged by modeling the type of systems desired, to promote Ministry buy-in at the centralized level. o How are communities proposing to sustain the efforts financially or in kind? At this time, communities do not feel confident in their capacity to sustain AYP, either financially or in kind, without support from USAID, the MOE, or another donor agency. AYP staff are in the process of developing and implementing a transition plan to encourage strategic thinking among communities about how they might continue ABE classes with their own resources; this process is still in its early stages and the planned series of meetings to undertake this planning had not begun, so communities did not have any feedback on its potential efficacy. Financial resources exist, either in the form of micro-payments, in-kind services to facilitators to compensate them for their time, and/or revenues generated by the youth clubs through soap making and other short-term skills on which AYP trained them. However, youth clubs have full ownership of these financial resources and seemed hesitant to use them to fund ABE classes. There was a strong sense of donor dependency and more confidence that someone would “come along to help them” pay for ABE classes if USAID does not continue to fund them. Question 2: Youth Targeting  What strategies worked best for recruiting and retaining out of school youth? The most effective strategies for recruiting out of school youth to attend AYP ABE classes included having free classes; community publicity campaigns; and pro-active ABE committees, facilitators, administrators and learners having personal interactions with youth to get them to enroll. The most effective strategies for retaining learners in the AYP classes were pro-active ABE committees, facilitators, administrators and learners personally reaching out and encouraging youth to return to classes; gender targeting whereby female learners targeted other females who had stopped coming to classes; providing role models and success stories to motivate youth to return; and having an engaging and student centric curriculum that engaged the learners and allowed them to participate in class. It is important to highlight here that learners have different motivations for enrolling and attending the ABE classes. Many learners, (especially some female participants) wanted to be able to write their name and to just attain some basic literacy and numeracy skills, while not having the dedication or time to continue and complete the three levels. Nonetheless, the one common and dominant strategy to recruit and retain out of school youth was the offer of short-term skills and long-term livelihood trainings. Further, being an active member of the youth clubs and making and selling soap or vegetables from the youth clubs gardens was also a big incentive for learners to remain in the program. The offer of trainings was a strategy that addressed the 50 needs of out of school youth who not only wanted to learn, but more importantly, wanted to gain some skills to generate additional incomes or find jobs. Most, if not all, youth who enrolled in the ABE classes expected to get some short-term skills trainings (to make and sell soap for example) or long term livelihoods training (at LOIC). However, in reality these trainings were limited to a certain number of learners who were active youth club members or learners who came to classes regularly. Thus the short term skills and long term livelihoods trainings were also paradoxically a reason for some learners dropping out of the ABE classes. Especially when the learners were not offered any short or long term trainings immediately or when they found out that they have to attend classes regularly to be eligible to obtain them. Beyond the critical issue of trainings, however, it has to be re-emphasized that there are broader contextual, cultural and systemic issues and reasons that make learners leave (or not enroll in) the AYP classes. Some learners just want to learn how to write their name while others have to work, take care of their families and cannot commit to attend and complete a nine month ABE course. It is thus extremely difficult for any adult basic education program to come up with an effective model to address all the various factors that preclude youth from attending ABE classes. Therefore, even the most successful strategies that AYP employed were not fully adequate to address the retention of its learners.  What interventions were most successful in helping to create positive outcomes (i.e., completion of course-level 1-3, internship/employment, other livelihood interventions)? The main factors that contributed to a learner completing the ABE level he/she was enrolled in included their personal commitment, regular class attendance, engagement in classroom activities, support from families and communities, capable and engaged facilitators and a student centric curriculum. Some learners also just wanted to complete a level (especially Level 1) so that they could learn to write their name and gain some basic literacy and numeracy skills. However, the one dominant factor that contributed to learners completing a level (apart from personal commitment) was once again the offer of short-term skills or long term livelihoods trainings. Therefore, in communities where learners had obtained some short-term skills trainings and were active in their youth clubs (and were making and selling soap and/or planning and selling vegetables grown in the youth club gardens) they were also more motivated to continue studying and to finish the level they were enrolled in. Further, a smaller subset of learners who obtained long-term livelihoods trainings (at LOIC) were also motivated to complete their level as one requirement for availing of these trainings required them to regularly attend classes at their AYP school site. Question 3: Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training? A much greater percentage of AYP learners had an opportunity to engage in short-term skills training than long-term skills training and were organized in a much more accessible way. Short-term trainings “Some learners ask what jobs they will get after joining. We tell them that they will get some skills or livelihoods trainings, and then they are motivated to attend the classes.” - Male Facilitator, Level 1, Montserrado County 51 for PTIs were held in central locations surrounding each community for a 2-week period, and then participants returned to their communities and served as trainers for the other learners at their sites. Long-term skills trainings were held at the LOIC in two locations – one for Montserrado County and one for Grand Bassa County. This required participants to travel long distances to attend training, or in some cases, to relocate their homes permanently or on a long-term basis, to participate. This posed a challenge for many learners and caused some participants to drop out of the training program. Rural communities in general benefitted more from skills trainings that took into account local contexts. Soap making was a predominant short-term training that was provided in nearly all of the 147 sites. It was a strong and adaptive initiative in the context of Ebola that resulted in youth clubs raising a significant amount of revenue to support their site, but having other varieties of different skills trainings based on changing local demand would create more appropriate and substantial learning opportunities for participants.  To what extent did livelihood trainings lead to quality employment for youth? The project provided many training opportunities, but less attention was given to project graduates and facilitating their pathways to leverage these skills into employment or money-making endeavors. There were no clear or consistent pathways to employment from the long-term skills training. Short-term skills trainings, which were more often accompanied by provision of tools and materials with which to implement the activities, were more consistent pathways to income generation. The informal economy in Liberia is a significant source of employment and self-employment opportunities for youth. Formal employment, especially outside of major city centers, is scarce and sometimes nonexistent. The potential for small, informal businesses needs to be better understood and tapped to identify promising entry points for young people in the Liberian economy.  To what extent did learners improve their literacy, numeracy, and work readiness skills to date? The OLA and EGMA assessments carried out in 2012/2013 had statistically significant, albeit low, gains in the learners’ literacy and numeracy scores from baseline to endline. Even though the EGMA demonstrated small achievements across all subtests, they indicated very low numeracy skills of learners. The OLA results showed that AYP learners who had IAI had improvements in their overall scores; however, there were overall low achievements on the majority of subtests among AYP participants (both who got and did not get IAI) between the baseline and the endline. AYP made significant changes (after these assessments) by revising the Level 1 and 2 curricula, placing more emphasis in facilitators’ trainings on teaching literacy and numeracy, developing IAI instruction for all subjects in Level 1 and conducting closer monitoring of AYP classes. The forthcoming OLA and EGMA endline assessments (baselines were done in 2015) will thus be more informative about how the measures that AYP adopted contributed to learners improving their literacy and numeracy skills and, further, what gender gaps exist among the learners. Nonetheless, despite what the forthcoming OLA and EGMA endline results will reveal; the qualitative data gathered from the field visits and the learners testimonies themselves clearly demonstrate that if both male and female learners are motivated, attend class regularly, participate in class activities, engage in group learning, complete their workbook exercise and have competent and engaged facilitators, they do significantly improve their literacy and numeracy skills. 52 Question 4: Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? At the moment, there is a lack of quantitative evidence about the efficacy of the AYP curricula, which makes a definitive conclusion about the program’s value impossible. However, strong feedback from beneficiaries on a qualitative level indicates that the AYP curricula are effective learning tools to build literacy, numeracy, life skills and livelihood potential for out-of-school youth in Liberia. There is a high demand for this type of educational opportunity among the target populations and a strong desire among respondents for the project to continue beyond the current lifecycle. If additional data can be produced to demonstrate efficacy of the ABE curriculum, it may be useful in encouraging the MOE to bring ABE service provision to scale, or at least including it as a priority area in forthcoming strategic plans. The MOE has integrated the curricula into their current activities, and there are other NGO service providers with experience and capacity to support expansion of this model to other areas and communities. The livelihoods skills trainings are extremely popular and in even higher demand than the literacy and numeracy training. Youth have a strong motivation to develop their earning potential through skills training, and when provided with the knowledge and tools to better their economic situation, conduct these activities with enthusiasm and passion. There is an untapped potential to strengthen their contribution to the Liberian economy and to engage them in productive activities that support their integration into community development. However, lack of resources to implement skills (such as tools and other materials) can inhibit trainees’ ability to transition this knowledge into active employment in informal and entrepreneurial markets. The MOYS could play a stronger role in supporting livelihoods and vocational training and have the political will to be larger actors in this work. Women and youth in rural communities are eager for opportunities to expand their capacity and economic development but require conscious accommodation to adapt implementation to their needs and social contexts. AYP has identified specific and effective strategies to retain females in their learning programs, including skills training opportunities that meet their interests and needs. Rural communities require support to access training sites, including stipends to fund their transportation costs. 53 RECOMMENDATIONS Question 1: Sustainability  To what extent did capacity building efforts lead to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth? Through the end of the current project lifecycle, AYP should continue to engage the CEOs and DEOs in the community-based transition plan meetings. AYP should also work to share and integrate monitoring tools to strengthen the capacity of interested individuals among the county and district staff; there are some extremely bright and motivated individuals in these positions who would welcome an opportunity to expand their capacity in this area. AYP should also continue to liaise with the MOE Central Administration to engage them in the transition work as much as possible to encourage their involvement and support for sustainability building efforts. Beyond the current project, USAID should continue to engage with MOE to encourage provision of ABE on a larger scale. While the lack of political will and/or reluctance to commit resources to this work serves as a barrier to bringing ABE service provision to scale, the immense scope of the need and the potential impact for good (economic growth) or bad (political instability) is a huge motivating factor in addressing this issue. Despite their limitations, the MOE is an essential partner in responding to this challenge and should be engaged to the greatest degree possible in capitalizing on the potential contributions of young people to their county’s economy and society. USAID should also continue to explore opportunities to expand the ability of the MOYS staff to collaborate more extensively with the MOE and donors and to encourage enhanced synergies between the MOYS and the MOE. The silo-ing of sectoral work and service provision that occurs between line ministries is particularly challenging for youth work, which by nature is cross-sectoral and requires collaborative and integrated processes. The MOE and MOYS should work to integrate the management of formal and informal vocational training centers in order to enhance the quality of service provision in both types of training facilities, to improve youth livelihoods opportunities for all. USAID should continue to develop and support provision of ABE teacher training through Stella Maris’ national curriculum and the rural teacher training institutes (RTTI), especially supporting the development of female teachers and teachers with stronger ties to rural contexts. Expanding the range and depth of the ABE teaching corps, especially among individuals who relate more closely to their learners, would enhance their ability to engage learners more deeply and to work to find solutions to issues that inhibit learner retention. In addition, developing a cadre of trained professionals to support the institutionalization of ABE within the MOE’s offerings is an essential step in bringing the project to scale. Finally, AYP’s NGO partners, LIYONET and NAEAL, should be provided all learning tools and information about ABE that emerges from the AYP Learning Agenda. They should also be involved in community transition planning in the communities where they are operating, especially since both organizations plan to continue providing ABE beyond the end of AYP’s project lifecycle. o To what extent were the Government’s expectations and policy needs met with AYP? The relevant MOE staff that has been engaged with AYP to date expects to be included in transition activities, and/or kept informed of AYP’s work through the end of the project period. Going forward, USAID and other donors should ensure that the MOE is consulted and integrated into the design phase 54 of projects through high level meetings, roundtable discussions, and working groups in order to ensure that project design is aligned with the MOE’s expectations and in order to enhance their engagement and buy-in for future activities. While this does pose a challenge in light of the high turnover rate and need for continued engagement, it is an essential element of their feeling of inclusion. The development of the AYP ABE curriculum was a very positive effort as it involved the MoE and various other international and national stakeholders in the TWG. This type of partnership and collaboration should continue for any future curriculum development for ABE. The AYP ABE curriculum should also continue to be used in any future ABE programs and as a resource for professional development for ABE teachers. Beyond this, the expansion of ABE to a greater scope of implementation depends upon projects’ and donors’ ability to identify and develop a strong champion for ABE within the MOE who can push it forward as a policy priority, perhaps a Director with influence who is not a political appointee and may have greater longevity. Without this, ABE is dependent on the will of the current MOE administration to support it; given the high turnover rates with the MOE staff, having a strong champion could help ABE weather the ever-changing storm of priority shifts within the Ministry. o How likely will the Government be to use the ABE materials in the future? Regardless of when AYP ends, a key element in USAID’s success in bringing ABE service provision to scale is to determine the Ministry’s ability and willingness to accredit or license ABE programs and identify the roadblocks to this process, including the Ministry’s process to offer accreditation. According to AYP staff, the Ministry itself is not clear on how to define and implement this process and needs support to outline a working procedure for it. Therefore, supporting them to a successful resolution of this issue could pave the way for external service providers to pick up a larger portion of the high demand among youth beneficiaries for ABE opportunities and create more uniformity in ABE service provision. In addition, USAID should continue to encourage the MOE to provide certification for their ABE learners, and to recognize AYP’s certifications in order to streamline AYP learners’ entry into conventional schools. Without hard data on the number of AYP graduates who transitioned to the conventional system, it is difficult to identify how pervasive an issue this is. Yet considering the number of respondents who requested completion certificates, the lack of systematic certification does affect their motivation and expectation to achieve future opportunities after completing AYP. Similarly, AYP should provide report cards for learners who complete a level so they have a record of their achievements. Many learners wanted to get report cards or progress reports after they pass a level as they can show these to their families and friends. These reports would also motivate the learners to perform better in classes. o How are communities proposing to sustain the efforts financially or in kind? For the remaining implementation period, AYP should continue its plans to provide in-depth support to communities to develop strong transition plans. They should engage in a collaborative partnership with the MOE Central Administration, CEOs, and DEOs, to encourage their involvement and support for this process. All of these actors are going to be key players in identifying strategic means to continue to provide AYP in the absence of USAID and/or MOE funding. This transition plan should include meaningful, practical and creative plans to address issues such as teacher stipends, materials production, and maintaining solar lighting. It should also consider enhancing the role of the ABE Committees in overseeing ABE activities within their communities, including managing funds to support its provision. 55 Finally, the transition plan needs to identify/develop a strategy to test learners on their skills mastery if the activity is to have any tangible certification or completion outcomes, which are extremely important to learners and their motivation to attend. Beyond this period, USAID should continue to encourage the MOE to institutionalize the service provision of ABE. They should also continue to develop the capacity of NGO service providers to provide ABE, in order to fill in the gaps in service provision and to provide a wider variety of ABE provision models that meet the needs of various communities. USAID should also work closely with other donors that are planning to implement similar projects, such as the EU, to leverage the investment made in AYP to expand its reach to other beneficiaries in need. Question 2: Youth Targeting  What strategies worked best for recruiting and retaining out of school youth? In the remaining time of AYP, the project needs to implement its community transition plan and aid the ABE committees in helping to think through how they will keep their structure, continue to exist and carry out their responsibilities. This way, the committees can sustain the retention of learners. AYP also needs to provide another round of trainings to ABE committee and youth club members, facilitators and administrators on recruitment and retention strategies, while also ensuring that female participation among ABE committee and youth club members is especially robust. In the future, if MoE or any USAID-funded program is to provide ABE to out of school youth, then one key issue that has to be addressed is to establish pathways for a larger number of learners to get short￾term skills and long-term livelihoods trainings. To continue a key strategy to recruit and retain learners, better pathways need to be established to assist a greater share of learners in obtaining these trainings. The provision of micro-credit loans (through VSLAs, cooperatives, matching loans, etc.) also has to be explored so that trained learners can access financial opportunities to start their own small businesses and ventures. Greater synergies will thus be required in the future so that the Education and Economic Growth teams at USAID can work together with multiple ministries such as the MOE and the MOYS to avail of training, micro-credit loan, TVET etc. opportunities, which are currently overseen by different divisions at USAID and by separate Liberian ministries. Efforts will also have to be made to work with private sector partners on a much larger scale to expand the provision of applied learning opportunities such as internships, on the job trainings, apprenticeships etc. so that learners have stronger livelihoods skills which are relevant to the market needs in Liberia. Some other recommendations that need to be considered include:  Data needs to be collected on those learners who return to ABE classes, thereby providing more formal information on the reasons for their return, and on what specific strategies motivated them to do so. AYP can still collect this data from learners who have returned to the ABE classes; and any future project should incorporate this topic as an output indicator. “There has to be more of an emphasis on skills trainings. After a learner gets basic literacy and numeracy and some training, they need to go to the next level. You cannot offer a similar project here that offers the same thing. You need to get them to the next level of work where they can use their training in a job or get some other vocational training.” - Male Facilitator, Level 3, Grand Bassa County 56  Formal studies need to be conducted that examine the effectiveness of incentives such as trainings and job linkages with ABE classes, so that there is clear and valid data to support future programs. Any future ABE project then should also collect this data from enrolled learners who proceed to get trainings or job linkages.  More tracer studies need to be conducted that follow the trajectory of ABE learners who got short-term or long-term skills and livelihoods trainings.8 AYP’s Learning Agenda is going to conduct a tracer study on this topic. Further, any future ABE project should incorporate conducting tracer studies as part of its overall M&E efforts.  Research needs to be carried out to examine (i) how projects that provided stipends to out of school youth to attend ABE classes and (ii) projects which charged youth a small fee to attend ABE classes are feasible and useful for the Liberian context. USAID needs to examine this topic further and determine if it will be beneficial to conduct this research study.  What interventions were most successful in helping to create positive outcomes (i.e., completion of course-level 1-3, internship/employment, other livelihood interventions)? For the remaining duration of AYP there has to be a consistent effort in the community transition planning to provide for mechanisms that will allow ABE committees to continue so that they can support the AYP schools and learners to complete their levels. Further AYP also needs to ensure that all sites have facilitators’ guidebooks and learners’ workbooks and provide another round of refresher ABE training for facilitators and administrators. Any future ABE project (either USAID-funded or MOE-led) to achieve positive outcomes for learners completing a ABE level needs to provide free learning (or at a very low cost); make use of the AYP student centric curriculum which has been well received by learners, facilitators and administrators; have trained facilitators and administrators who are capable and engaged; and support the building of ABE committees which support the schools and learners. Another issue pertains to measuring learners’ expectations and motivations for enrolling in ABE classes. This data needs to be collected so that any future project is able to discern not only the motivations of learners’ to attend ABE classes, but also to address whether these motivations were met, if they changed and how pertinent they were for learners to complete an ABE level. The integrated model of AYP encompassing ABE and skills and livelihoods trainings is optimal, but significantly more pathways are needed for trainings and job connections. This means offering a greater number of short-term skills and long-term livelihoods trainings, along with employment linkages. Further, there has to be adequate diversification in the skills and livelihoods trainings to meet local and national demands. 8 The AYP Learning Agenda is doing a mini tracer study on this topic. 57 Question 3: Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training? For the remaining period of AYP’s implementation, the activity should continue to support the activities of PTIs and youth clubs to provide skills training for learners. This should include the extension of short-term skills training from PTIs to additional members of the community as needed and desired by the communities. AYP should also support the continuation of GYB classes in communities where the activity is strong and has engaged participants, as this element was not allowed the full amount of time needed for communities to complete the curriculum and its impact was very strong in rural areas where farming has a major livelihood potential. For future interventions of a similar nature, interview learners early on to ascertain their expectations for participation, and work to meet them. The current activity is resulting in an untapped demand for on-the-job training (OJT) due to issues of quality and accessibility. Explore creative means to overcome barriers to youth’s participation in long-term skills training, including issues of accessibility and related challenges of taking trainees out of their local communities for long periods of time (especially young women). This is a crucial element in supporting their engaged participation in the training, and thus facilitating their enhanced livelihood skills. Liberia’s TVET system needs strengthening. Facilitate more effective pathways to TVET training through other service providers, funding mechanisms, and engagement with line ministries such as the MOYS in order to make skills training opportunities accessible and effective for youth. Support the development of a strong, responsive curriculum, teachers who have the expertise to deliver it, and up-to-date equipment for learners to use in their classes. Explore innovative livelihood opportunities that take advantage of skilled labor niches and expand the range of skills and services that are offered by the Liberian labor market. Ensure that skills trainings are specifically relevant to the local context and economies, so that these investments are well-utilized and effective support for the learners. Purchase the needed materials that were given to the service provider and the learner to kick start the training program.  To what extent did livelihood trainings lead to quality employment for youth? Before the end of AYP’s lifecycle, the activity should undertake a strong tracer study to determine the impact of AYP’s livelihood skills training on learners’ employability. Without this information, it is impossible to determine how effective the activity was, and this information is in-demand with a wide variety of stakeholders. If this cannot be done under the current activity, USAID should consider undertaking the study through another mechanism in order to ensure the information is captured while needed data are still fresh and available. Looking forward to future interventions, data on the number or percentage of beneficiaries who achieve targeted outcomes should be collected systematically throughout implementation, in order to make necessary adjustments during project lifecycle to maximize desired outcomes. An activity should not come to the last year of implementation lacking strong outcome-level performance management data reported to USAID. The project needs to be more proactive in this direction to track graduates and their outcomes, and make sure they are linked to institutions that could assist them after their participation in AYP ends. 58 Provision of long-term skills training should be accompanied by tools and materials needed to undertake those activities - either as a direct element of service provision, or through linkages with other financing opportunities such as VSLAs and microfinance projects. Learners do not just need livelihoods trainings; they also need small capital to start micro-enterprises. In addition, linking LOIC graduates to other organizations or opportunities that have similar opportunities would be a strategy to promote employability for these individuals. Explore models of service provision that incorporate innovative stipends for learners, which will reward engaged participation and also facilitate pathways to informal employment opportunities post￾participation. This might include stipend money held in trust for learners dependent upon their completion of the course, at which time they are paid for the purchase of goods and materials to start a small business.  To what extent did learners improve their literacy, numeracy, and work readiness skills to date? The OLA and EGMA endline assessments will be conducted in June 2016 and, as far as possible, the project should also identify the assessed AYP learners based on their attendance in class i.e. regular learners versus irregular learners (who did not come to ABE classes regularly). This will allow the project to determine the correlation between attendance in ABE classes and improvements in literacy and numeracy scores. As the OLA and EGMA tools are only administered to a small segment of the learners, AYP also has to rely on other data sources to demonstrate the learning gains of out of school youth. AYP needs to quantify completion tests results, classroom observations of learners and facilitators and completed workbook exercises of learners, while also collecting and quantifying data pertaining to individual learner attributes such as attendance and engagement in classroom activities. This supporting data will then aid in providing a more holistic picture of the learning environment of AYP and the learning gains and outcomes for out of school youth. Any future ABE project in Liberia also has to have standardized learning assessment tools and methods. Therefore AYP also needs to, as part of its community transition planning, provide ABE committees information, options and standards on how youth’s learning can be assessed in the future. Further, the MOE also needs to establish learner assessment standards and tools to determine literacy and numeracy competencies that will align with the ABE curriculum they offer, as incorporating and administering OLA and EGMA will be challenging for the MOE. Other Recommendations Additional recommendations emerged over the course of this evaluation pertaining to the learning component of AYP:  As the current youth club gardens in the AYP communities have very basic tools, it is hard to fully demonstrate the learning from the GYB curriculum. Thus, setting up a farm in a convenient central area where GYB learners can go (in batches) and practice and see the implementation of their learning will be very beneficial. In such a central farm the learners can also do higher level research on agricultural topics.  The Learning Agenda will include many reflective pieces on the learning component of AYP. AYP thus has to ensure that public dissemination events/forums are held and short and long 59 briefs are developed and disseminated to various stakeholders (including the MoE at all levels) to raise awareness of what works and doesn’t work in providing and improving literacy and numeracy (along with life skills and work readiness skills) of out of school youth.  The Learning Agenda outputs are crucial and useful for current partners and other international and local NGOs that can provide ABE to youth in the future. Further, they will provide the MoE with relevant evidence to inform ABE policy.  Any future ABE project that offers skills and livelihoods trainings needs to be explicit about what the trainings will cover and what they will lead to, thereby managing the expectations of the out of school youth that enroll in the program. Question 4: Cross-Cutting Analysis  Given the remaining time, scope, and budget of AYP, what are the most effective interventions that can be built upon in the remaining time of AYP and also looking forward to future interventions to support youth in Liberia? If AYP is granted the no-cost extension, their focus for the final period of implementation should be to support an effective and efficient transition plan for the AYP communities. As stated above in Question 1, this should include to the greatest extent possible the full range of MOE partners to encourage their support and engagement in this process. AYP should also undertake the planned research activities that will provide a fuller picture of their impact in building beneficiary capacity in areas like: institutional and organizational capacity building and strengthening of literacy, numeracy, life skills and livelihood opportunities. Finally, AYP should work in close collaboration with any local or international organizations that are planning to continue or start to use the ABE curriculum for future work, to share knowledge and lessons learned through this project and in order to enhance the quality and efficacy of future service provision. After AYP ends, USAID should continue to explore opportunities to expand the provision of literacy, numeracy, life skills, healthy actions, and livelihoods skills training to a wider range of Liberian counties and communities. Any new project developed to address ABE among youth populations in Liberia should expand the scope of and ensure learner accessibility to skills training, especially for rural communities and women. This work should adopt an integrated approach to youth development, which will require cross-sectoral cooperation that is a challenge for many institutions. This is a necessary element in the success of these endeavors, as youth work requires the integration of educational, economic livelihoods (including agricultural work), and health sectors in order to provide holistic support for the development of young people. These approaches should include pathways to enhance continuity of services for beneficiaries after they complete each step along the way. For example, an educational project should be able to transition participants into an economic growth project that supports young entrepreneurs; life skills projects should include healthy lifestyles curriculum including sexual and reproductive health topics, etc. This approach should be integrated at both the donor and governmental levels (i.e., line ministries) to promote a more cooperative and integrated system at all levels of developmental service provision. This also goes across donor agencies: USAID should continue to work in close collaboration with other donors who are designing and implementing ABE projects of a similar nature to leverage the strong 60 investment made through AYP, to continue to utilize its assets (curriculum, solar lights, trained facilitators and facilitator training programs). Collaboration with these institutions to support a greater understanding of the needs of out-of-school learners will ensure impact beyond the six counties that USAID implements and lead to the development of stronger and more complementary range of projects that meet the identified needs. This is especially true for females and rural communities, if the activity can offer more accessible training locations and topics that ensure relevance to rural contexts. 61 ANNEXES 62 ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Performance Evaluation of USAID’s Advancing Youth Project Evaluation Purpose: USAID/Liberia is commissioning a performance evaluation of the Advancing Youth Project (AYP) in order to document lessons learned and to inform the design of future education programs which focus on USAID’s Education Strategy--Goal 1 (literacy) and Goal 3 (access). Summary of program: Project Title: Liberia Advancing Youth Project Project number: Cooperative Agreement: AID-669-A-11-00001 Project dates: October 24, 2011-October 23, 2016 Project funding: $39,899,604 Implementing partner: EDC Contracing Officer’s Representative (COR): Mardea Nyumah Active Geographic Regions: Montserrado, Margibi, Bong, Lofa, Nimba, and Grand Bassa The goal of Liberia’s AYP is to enhance the capacity of governmental and nongovernmental organizations (NGOs) to provide increased access to quality alternative basic education (ABE) services, social and leadership development and livelihoods for youth and young adults, ages 13-35 who are un-schooled or out of school and have no or marginal literacy and numeracy skills. The project goal is supported by two strategic objectives:  Objective 1: Capacity and Sustainability: Ministry of Education and community based organizations (CBOs) have the capacity to support and manage alternative basic education programs in Liberia with efficiency, effectiveness, and equity  Objective 2: Access and Quality: Youth have access to clearly defined and relevant education and livelihoods pathways AYP’s Results Framework (figure 1), explains how the first objective focuses on building the capacity of government institutions, as well as the NGOs and CBOs, which will sustain the ABE program and the enhancements made under Advancing Youth. Intermediate Result 1 under this objective is: 1) increased institutional capacity to plan, budget, deliver and measure ABE for youth. The outcome indicators focus primarily on measuring institutional and organizational capacity and the output indicators focus on the policy, curriculum, textbooks and learning materials (TLM) and other system resources and inputs critical to implementing the ABE programming. Key activities under this objective include: planning and engaging stakeholders in ABE systems development; drafting and revising ABE policy(s); developing and distributing TLM; training institutional and organizational officials; developing assessments to measure outcomes; and utilizing action research to support ABE system strengthening. The second objective focuses on ensuring access and quality of ABE programming and livelihood for those youth of various ages and backgrounds that were left out of traditional formal education systems. Intermediate Results 1 and 2 under this objective are: 2) increased access to relevant basic education and life, work, and technical skills training for out-of-school youth and 3) increased access to sustainable livelihood pathways for targeted youth. A diverse range of outcome and output indicators are used to measure education, economic growth and health related activities and interventions. Key activities include: developing three levels of curriculum in literacy, numeracy, life skills and work readiness; training facilitators to deliver alternative basic education; providing youth with work-based learning opportunities, vocational and technical skills 63 development, and entrepreneurship training; forming youth clubs and local alliances to support youth education and enhanced livelihoods; and developing private-public partnerships. Using tools (protocols and procedures developed under Objective 1) for assessing change in literacy (reading), work readiness, and institutional capacity are also critical to ensuring youth are obtaining relevant skills and that activities are achieving results. These activities treat age as a critical factor in learning, while also expanding access to learners through flexible scheduling models, and distance learning tools through use of technology. Advancing Youth will collaborate with MoE and other partners/stakeholders at all steps (design, implementation, and assessment) to ensure that capacity is sufficiently built under both strategic objectives and ultimately leads to the project goal. By the end of the project, the project will have achieved the following key results:  80 percent of sampled ABE partner institutions and organizations have increased capacity  Increase in reading skills by end of year 2 (Level 2) and end of primary equivalency (Level 3) for sample youth  Revised and updated ABE policy guidelines and three levels of ABE curriculum and materials completed  360,879 learning materials developed/acquired and distributed  2 standardized learning assessments developed  316 principals and other administrators/officials trained in pedagogy and management of ABE  2,913 facilitators trained pedagogy and content of ABE curriculum  16,000 youth reached AYP will ultimately contribute to USAID’s 2011 Education Strategy Goal One of improved reading skills for 100 million children in primary grades by 2015 and Goal Three of increased equitable access to education in crisis and conflict environments for 15 million learners by 2015. 64 Figure 1: Results Framework AYP has already produced a number of documents which will help the evaluator conduct the evaluation. USAID and EDC will provide the evaluation team with a package of briefing materials to review which will include some of the following documents:  AYP Work plan  Annual implementation plan  Activity quarterly reports  Annual Report  Activity M&E Plan/ PMP  Out of School Literacy Assessment Result 3: Increased access to sustainable livelihood pathways for targeted youth Result2: Increased access to relevant basic education and life, work, and technical skills training for out-of￾school youth Result 1: Increased institutional capacity to plan, budget, deliver and measure Alternative Basic Education for youth Sub-result 3.3 Young (especially women) are linked to relevant and realistic livelihoods skills opportunities Sub-result 3.1 National and county-level alliances for youth employment are established to provide work-based learning, skills training and entrepreneurship opportunities for targeted youth Sub-result 2.1 Strategies for recruiting and retaining youth, especially young women, are planned and implemented Sub-result 2.3 ABE students have improved literacy, numeracy, work readiness, health and life skills Sub-result 2.2 ABE facilitators (both government and non government) have the capacity to effectively teach all levels of the ABE curriculum Sub-result 1.3 A comprehensive ABE curriculum is developed and implemented Sub-result 1.2 Selected local institutions will have the capacity to partner with the MoE to deliver quality ABE instruction, and/or promote entrance and persistence in ABE Sub-result 1.1 MOE is able to set national ABE policies, prepare annual implementation plans and budgets, and train, support and monitor ABE teachers, based on evidence of best practices Sub-result 3.2 Selected NGOs have the capacity to deliver livelihoods training programs, based on assessed local market needs Strategic Objective 1: Capacity and Sustainability: MoE and CBOs have the capacity to support and manage alternative basic education programs in Liberia with efficiency, effectiveness, and equity Strategic Objective 2: Access and Quality: Youth have access to clearly defined and relevant education and livelihoods pathways Project Goal: Enhance the capacity of governmental and nongovernmental institutions to provide increased access to quality alternative basic education services, social and leadership development and livelihoods for youth and young adults, ages 13-35 who are un-schooled or out of school and have no or marginal literacy and numeracy skills 65  Out of School Math Assessment  AYP Learning Agenda  Youth employability Study  Labor Market Assessment  Institutional Capacity Assessment of Ministry of Education units  Organizational Capacity Assessments of local partners  Success Stories  Studies conducted by other organizations around youth in Liberia  Other Documents as requested Evaluation Questions: Sustainability  To what extent did capacity building efforts led to sustainability in the Ministry’s ability to plan, budget, deliver and measure ABE for youth?  To what extent were the Government’s expectations and policy needs met with AYP?  How likely will the Government use the ABE materials in the future (confirm that the Ministry has provided accreditation and certification for the alternate learning program and ABE)? Youth targeting  What strategies worked best for recruiting and retaining out of school youth?  What interventions were most successful in helping to create positive outcomes (i.e. completion of course-level 1-3, internship/employment, other livelihood interventions)? Outcomes  To what extent did the program succeed in linking youth to “relevant and realistic” livelihoods training?  To what extent did livelihood trainings lead to quality employment for youth?  To what extent did learners improve their literary, numeracy, and work readiness skills during the life of the project? This will be informed by EDC's own Out-of-School Literacy Assessment (OLA) and the Early Grade Mathematics Assessment (EGMA), but an independent review of these materials will help inform the Mission and the GoL to become more effective at improving learning outcomes for young adults in the classroom. Cost Effectiveness  What was the cost per beneficiary who advanced to Level 3, and how does this compare to similar interventions? Design and methodology: USAID/Liberia is requesting a mixed-methods performance evaluation that will enable evidence-based answers to all evaluation questions. The following methods could be used:  Document review  Cost benefit analysis  Focus group discussions  One on one interviews with key stakeholders  Case studies 66 Social Impact will develop a design/inception report which will include the evaluation methodology, team composition, stengths and limitations of proposed methodology, roles and responsibilities, work plan, analysis plan, and instruments for USAID’s review and approval. Evaluation Team composition: The evaluation team will be composed of four individuals: team leader, 2 senior technical experts, and a logistician.  Team Leader/Senior Evaluation Specialist: S/he will be responsible for coordinating the activities of the evaluation team, and have the authority to make budgetary and programmatic decisions regarding the evaluation. S/he will serve as the main point of contact between USAID and the contractor’s headquarters. The Team Leader will approve the final evaluation design, oversee the development of evaluation instruments, integrate the findings of different team members and coordinate the preparation of the final reports. The Team Leader should have at least ten years of experience in the administration of multi-faceted education projects in developing countries - preferably in West Africa. S/he should have experience in managing multi￾disciplinary teams and developing and conducting qualitative evaluations, the ability to conceptualize and structure evaluation activities and write clearly and concisely. An MA or PhD in education administration, planning, economics of education or similar field is required. Experience working in post conflict and transition settings is preferred.  Education Specialist: The composition of the two subject matter specialists will depend on the final evaluation questions, but one should be a methodologist (evaluation specialist) and the other a sectoral specialist (ALP). The Education Specialist will hold an advanced degree in Educational Research and/or Statistics and have extensive experience in evaluating educational programs. S/he will have at least 8 years of experience working with educational projects in developing countries, preferably in post-conflict environments.  Liberian Education Context Specialist(s): The local consultant shall have experience and knowledge about the education context in Liberia, particularly on youth, work force development and literacy and numeracy programs.  Logistics Coordinator: S/he will serve as the main logistical coordinator for the performance evaluation, and work with local partners to plan travel, data collection, interviews and assessment activities as required. S/he will work with the Team Leader to review, edit, and format the final report of the evaluation, prepare it for production, supervise the production and distribute it to USAID/Liberia. USAID expects that this role will be covered by LSA staff. The Team Leader is required to have demonstrated expertise in evaluation methodology. Collectively the team members must have experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in Liberia, all team members are required to familiarize themselves with USAID’s Evaluation Policy, with USAID’s publication outlining a good evaluation report, and with USAID’s checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic, policy and educational context in Liberia. Scheduling: Work should be scheduled while school is in session, before AYP ends, and before the rainy season starts. The entire evaluation would be carried out over 10 weeks, with 2 weeks of preparatory time, 4 weeks in the field, 2 weeks of analysis and 2 weeks of report writing and editing. The evaluation team should work six-day work weeks. 67 Estimated level of effort Team Leader Evaluation Specialist Education Specialist Review background documents 2 2 2 Draft design report 3 3 2 Travel to Liberia 1 1 0 Mission inbrief .5 .5 .5 Pilot testing .5 .5 .5 Field work 20 20 20 Preliminary analysis 2 2 2 Mission outbrief 1 1 1 Travel from Liberia 1 1 0 Analysis 4 6 4 Submit draft report 5 3 3 Submit final report 2 1 1 Findings workshop 1 0 1 TOTAL 43 41 37 Evaluation deliverables:  Design/inception report  In briefing with USAID before beginning field work  Debriefing with USAID after conclusion of field work and before the analysis  Stakeholder findings workshop  Draft evaluation report  Final report—will adhere to PPL’s guidance on evaluation report requirements. The evaluation team shall incorporate USAID’s comments and submit final report to USAID in electronic format as well as printed and bound copes. Social Impact will submit the final evaluation report to the Development Experience Clearinghouse within 90 days of approval. 68 ANNEX II: EVALUATION METHODS AND LIMITATIONS EVALUATION METHODOLOGY Document Review During the first week of the evaluation, the team conducted a thorough review of background documents related to AYP, including periodic program reports, contractual documents, research studies, and other relevant documents to contextualize the research. These documents enabled the team to gain a deeper understanding of the project’s operations to date, to establish the evaluation design and data collection tools, and to inform the overall project assessment process, including conclusions and recommendations for future programming. These documents included the following: 1. AYP Cooperative Agreement and Program Modifications 2. AYP Staff Contact List 3. AYP annual work plans and Life of Agreement strategic plan 4. AYP M&E plan/PMP 5. AYP annual reports 6. AYP activity quarterly reports 7. AYP baseline survey report 8. AYP research and survey reports  Livelihoods and Work Readiness Report  Youth Livelihoods Assessment Reports 2013 and 2014  Labor Market Assessment  HealthyActions Final Evaluation 9. Studies conducted by other organization focused on youth in Liberia 10. Success Stories developed for AYP and other relevant communication materials 11. AYP performance indicator database 12. AYP site listings and contact matrix 13. List of AYP partners and short description of their scope 14. Relevant reports from other donor agencies 15. Relevant GoL policy documents and technical reports 16. Early Grade Mathematics Assessment (EGMA) report FIELD WORK After the background review phase, the team spent four weeks in the field conducting site visits and collecting qualitative data in order to gain a deeper understanding of AYP, the experiences of its direct beneficiaries, and the local implementation environment. During this time, the evaluation team visited 12 AYP sites, which were selected in three of the six counties in which the project is implemented, and included seven urban and five rural locations. The school sites visited by the evaluation team were selected through random sampling of a full list of all locations within the three target counties, combining the team’s desire to sample a mix of centralized and remote locations, with the need to balance general accessibility for the team. Accessibility issues included: availability of local accommodations nearby; timing of classes and safety issues of driving on poorly-maintained roads in the dark; and balancing the amount of time it would require the team to 69 travel to reach a destination with overall efficiency of the team’s travel arrangements. Table 1 lists the sites visited by county. Table 1. Field Sites Visited County Site Classification Montserrado County Grey D. Allison Elementary School Urban Kortu Town Community School Rural St. Paul’s Bridge Elementary School Urban Logan Town Central Public School Urban Nimba County Sanniquellie Elementary School Urban Gehwee Elementary School Rural New Yekepa Elementary School Rural Tondin Elementary School Rural Grand Bassa County Lowerharlandsville Public School Urban Little Bassa Public School Rural Benson River Public School Urban Cotton Tree Public School Urban Key Informant Interviews (KIIs) During the site visits, the evaluation team conducted a wide range of Key Informant Interviews (KIIs) with AYP project beneficiaries, including: AYP learners (current and completed, as well as those involved in the Youth Clubs); school administrators and facilitators; County Alliance members; ABE Committee members; LRC county office staff (including County Team Leaders, M&E staff, and YDLCs). In addition to the learning sites, the evaluation team conducted extensive interviews with AYP program staff and partner organizations, representatives of the Liberian Ministry of Education (MOE) and Ministry of Youth and Sports (MOYS), other donor agencies and USAID offices with relevant programs. These interviews were guided by semi-structured interview protocols that identified the key evaluation questions that needed to be answered through the interview. These semi-structured guides also identified probes to follow up on information related to key questions, in order to allow flexibility in the discussion and a natural flow to the conversation, while still allowing the evaluation team to follow up on any points of particular interest to the research questions. Overall, respondents associated with the learning sites were selected and identified based on their affiliation with the selected site. For a vast majority of the schools visited, the evaluation team interviewed most of the individuals associated with the site: the school administrator, one or more of the classroom facilitators, most of the members of the ABE Committee when present, and a vast majority of the learners. In a few cases, where the number of learners present was larger and/or space permitted, a sampling of learners would be called together for the focus group. The larger sample of learner focus groups were a change from the originally-planned methodology; while the team had planned to focus more on interviews with individual learners, rather than conduct a full focus group in each site, the situation on the ground dictated a different approach. The learning communities themselves indicated a desire to meet as a whole, and in five of the eight learner focus groups, all of the site’s learners participated. While this did make for large discussion groups, the evaluation team decided that it was important to honor the respondents’ wishes rather than insist on a 70 more textbook approach to data collection. This decision also resulted in a smaller number of individual learner interviews being conducted, as most respondents had already been sampled in the focus groups. Table 2 below details the number of KIIs undertaken by stakeholder group. Table 2. Number of Key Informant Interviews Targeted and Reached Respondent Group Total targeted Total Completed AYP Head Office Staff 10 10 AYP partners 6 4 Government of Liberia 6 8 Other donor programs and USAID offices 5 2 ABE Leaders and Community Partners 18 19 ABE Learners 9 3* Employed and unemployed graduates of AYP 6 9 Focus Group Discussions (FGDs) The evaluation team also conducted focus group discussions (FGDs) with select groups of AYP stakeholders focusing on one or many of the evaluation questions. The decision to apply focus groups for select communities was made in order to accommodate a wider range of voices from stakeholder groups that often work as a team (e.g., the ABE Committees, County Alliances, and Youth Clubs). Table 3 below lists the respondent groups that participated in focus groups, along with the original target and the completed sample size per respondent category. Table 3. Number of Focus Groups Targeted and Reached Respondent Group Total targeted Total Completed ABE Committees 3 4 County Alliance members 3 2 Youth Club members 3 2 ABE Learners 3 8* * Planned individual interviews with AYP learners were replaced with focus groups in response to field contexts Data Analysis and Reporting Data collected during the KIIs and FGDs were captured in note form, typed up and circulated to the other team members for additional inputs. These final versions were analyzed using an inductive approach, gathering all of the data before engaging in a day-long group discussion to identify and debate amongst the evaluation team members any themes and patterns within the data. This process informed initial development of key findings, conclusions and recommendations that make up the bulk of this report. From there, the evaluation team members worked individually to analyze qualitative data using a keyword sorting system to group respondents’ inputs into main categories that enabled a systematic analysis of all data. Finally, these processes culminated in the development of this performance evaluation report. 71 DATA LIMITATIONS Limited Access to Data – As mentioned above, the evaluation team had limited access to AYP’s learner data and other information that the project deemed too sensitive to release to an external evaluation team. For some output indicators, the information was not reported to PIDS in aggregate either. For the quantitative data that are available in PIDS or through top-line reports, this evaluation report does include summary analysis, and in some cases, substitutes qualitative findings for the quantitative evidence of outcomes. While this approach does provide sufficient information to make informed assessment of the project’s efficacy in some areas, it does limit the evaluation team’s ability to make a definitive assessment of particular project elements; this is especially pervasive in evaluating the performance of the institutional capacity development work and the efficacy of the project curricula to increase learners’ knowledge gains. In addition, it was difficult to ascertain exactly what are the motivations and political will of the MOE to institutionalize ABE into their strategic plan. Although a number of Ministry officials were interviewed during the qualitative data gathering process, there was no clear answer on what will be the policy going forward and how the Ministry plans to integrate ABE into their planning process. Many times, answers contradicted each other when attempting to triangulate data across stakeholders; thus, the evaluation could not take into account any future planning for ABE and took the current status at face value in making recommendations for AYP. Sampling Strategy – While the evaluation team endeavored to select a randomized sample of respondents from each category, in some cases, AYP staff were called upon to facilitate contacts on behalf of the evaluation team. This was particularly influential in the identification and facilitation of meetings with learners who had completed the program. In addition, school administrators facilitated access to respondents such as the ABE Committee members, some facilitators, and learners. In addition, communities often organized themselves into mixed respondent groups (e.g., administrators and facilitators, facilitators and learners, etc.) which seemed to be their preferred method of meeting. However, it is possible that in some cases, respondents felt less free to speak openly in the presence of others. The evaluation team felt it was best to respect the communities’ Scope and Timeframe of the Evaluation and Project Implementation – This performance evaluation was conducted over a 10-week period, which allowed for two days of background document review, which limited the team’s capacity to review the volume of project documentation that was available. In addition, the four weeks allotted to field work allowed the team to visit only 12 of the 147 AYP sites, and in three of the six counties in which AYP operates. This gave the evaluation team a snapshot of the vast range of sites in which the project is being implemented. In addition, the AYP project itself faced challenges in timing of implementation, most especially in the fourth year of the project, when the Ebola crisis severely limited the scope of activities that could be implemented. Schools closed for some six months, and most activities that required in-person meetings ceased. This not only affected the project’s outputs during that period, but also limited the capacity of the project to conduct other activities, such as data gathering and research agenda items. Over-surveying of respondents – EDC has done a vast number of studies and assessments of the work conducted under AYP, so the respondents are used to being contacted for information on the project. However, there could be some respondent fatigue that would influence the validity of the data. In addition, AYP’s plans to undertake internal studies influenced their willingness to share data with 72 external evaluators – especially for the OLA and EGMA data, for which they were currently undertaking an endline test, with results forthcoming. Incentivizing Respondents’ Participation – The evaluation team experienced some challenges in hosting focus groups, and ensuring that respondents would attend without any incentives to compensate them. While no monetary incentives were offered, the evaluation team did provide light refreshments for the respondents, which could have influenced the respondents’ feedback. In order to minimize the impact of this tendancy, the refreshments were distributed at the end of the discussion period. 73 ANNEX III: DATA COLLECTION INSTRUMENTS Advancing Youth Project (AYP) Performance Evaluation KII with Ministry of Education (MoE) Officials Name: Title: County: Date: Gender: M F Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your office/Ministry in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us, and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion. Start with an overview of the evaluation’s purpose and objectives 1) What are your responsibilities at the MoE? 2) What has been your involvement with AYP?  Department level? 3) Do you feel that the Government’s expectations and policy needs were met with AYP?  What expectations and policy needs still need to be met? 4) Are ABE program delivery models flexible and realistic strategies to meet the program’s goals?  school/community governance structures (ABE committees, county alliances, youth clubs etc)  organizational capacity of CBOs/NGOs to partner with MoE 5) Do you feel that AYP’s Program Quality Standards (PQS) successfully address ABE service delivery in the country? 6) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Do you think the MoE will be likely to the curriculum/materials in the future? If so, in what way? 7) To what extent do you think AYP’s capacity building efforts are sustainable?  What type of transition plans exist at the MoE? Are they feasible?  What will be the barriers to a successful transition to a sustainable model once USAID funding ends? 74  What challenges would external partners face to provide ABE services?  Do you know if there are other models of programs that transitioned to self-sustained activities? 8) Are physical contributions like the solar lights that USAID has made to the school sustainable? 9) Do you anticipate that the MoE will provide accreditation and certification for alternative basic education programs?  For standards for ABE providers?  For ABE teachers? 10) Which AYP strategies/interventions do you feel have worked best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 11) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 12) Do you feel that AYP built the capacity of the teachers/facilitators and administrators?  How is the MoE supporting them? What are the challenges? 13) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 14) Do you have recommendations for AYP going forward? 15) Do you have any feedback or suggestions about other issues or questions that you would like to gain from this evaluation, within the scope of work we outlined in the beginning? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 75 Advancing Youth Project (AYP) Performance Evaluation KII with County Education Officers/District Education Officers Title: County: Date: Gender: M F Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your office in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion. 1) What are your responsibilities at the MoE? 2) What has been your involvement with AYP?  Department level? 3) Do you feel that the Government’s expectations and policy needs were met with AYP?  What expectations and policy needs still need to be met? 4) Are ABE program delivery models flexible and realistic strategies to meet the program’s goals?  school/community governance structures (ABE committees, county alliances, youth clubs etc)  organizational capacity of CBOs/NGOs to partner with MoE 5) Do you feel that AYPS’ Program Quality Standards (PQS) successfully address ABE service delivery in the country? 6) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Do you think the MoE will be likely to continue using the curriculum/materials in the future? If so, in what way? 7) To what extent do you think AYP’s capacity building efforts are sustainable?  What type of transition plans exist at the MoE? Are they feasible?  What will be the barriers to a successful transition to a sustainable model once USAID funding ends?  What challenges would external partners face to provide ABE services?  Do you know if there are other models of programs that transitioned to self-sustained activities? 8) Are physical contributions like the solar lights that USAID has made to the school sustainable? 76 9) Do you anticipate that the MoE will provide accreditation and certification for alternative basic education programs?  For standards for ABE providers?  For ABE teachers? 10) Which AYP strategies/interventions do you feel have worked best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 11) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 12) Do you feel that AYP built the capacity of the teachers/facilitators and administrators?  How is the MoE supporting them? What are the challenges? 13) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 14) Do you have recommendations for AYP going forward? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 77 Advancing Youth Project (AYP) Performance Evaluation KII with AYP County Staff (Team Leaders, YLDCs, ABE Coordinators) Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). 1) What are your responsibilities with AYP? 2) What has been your involvement in working with the MoE? 3) Do you feel that the Ministry’s expectations and policy needs were met with AYP?  What expectations and policy needs still need to be met? How could this be achieved? 4) Are ABE program delivery models flexible and realistic strategies to meet the program’s goals?  school/community governance structures (ABE committees, county alliances, youth clubs etc)  organizational capacity of CBOs/NGOs to partner with MoE 5) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Do you think the MoE will be likely to the curriculum/materials in the future? If so, in what way? 6) To what extent do you think AYP’s capacity building efforts are sustainable?  What type of transition plans exist at the MoE? Are they feasible?  What will be the barriers to a successful transition to a sustainable model once USAID funding ends?  What challenges would external partners face to provide ABE services?  Do you know if there are other models of programs that transitioned to self-sustained activities? 7) Which AYP strategies/interventions do you feel have worked best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 8) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 9) Do you feel that AYP built the capacity of the teachers/facilitators and administrators? 78  How is the MoE supporting them? What are the challenges? 10) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 11) Do you have recommendations for AYP going forward? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 79 Advancing Youth Project (AYP) Performance Evaluation KII with Subcontractor & Sub-grantee Representatives Title: Organization: County: Date: Gender: M F Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). 1) What has been your involvement with AYP? 2) Are you offering both literacy and numeracy training, and livelihoods development activities? Please specify which ones? 3) Do you collaborate with local community groups that support AYP? 4) Are ABE program delivery models flexible and realistic strategies to meet the program’s goals?  school/community governance structures (ABE committees, county alliances, youth clubs etc)  organizational capacity of CBOs/NGOs to partner with MoE 5) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Do you use any other curricula in your AYP classes? 6) To what extent do you think AYP’s capacity building efforts are sustainable?  What type of transition plans exist at the MoE? Are they feasible?  What will be the barriers to a successful transition to a sustainable model once USAID funding ends?  What challenges would external partners face to provide ABE services?  Do you know if there are other models of programs that transitioned to self-sustained activities? 7) Which AYP strategies/interventions do you feel have worked best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 80 8) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 9) Do you feel that AYP built the capacity of the teachers/facilitators and administrators?  How is the MoE supporting them? 10) Do you feel that working within the AYP structure has enhanced the capacity of your organization? If so, in what way? 11) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 12) Do you have recommendations for AYP going forward? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 81 Advancing Youth Project (AYP) Performance Evaluation KII with ABE Facilitators & Administrators Title: County: Date: Gender: M F Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). 1) W hat has been your involvement with AYP? 2) (For Facilitators) Which level of AYP classes do you teach? 3) Are ABE program delivery models flexible and realistic strategies to meet the program’s goals?  school/community governance structures (ABE committees, county alliances, youth clubs etc)  organizational capacity of CBOs/NGOs to partner with MoE 4) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Which modules do you feel the learners enjoy the most/least? Benefit the most/least in terms of knowledge gains? 5) To what extent do you think AYP’s capacity building efforts are sustainable?  What type of transition plans exist at the MoE? Are they feasible?  What will be the barriers to a successful transition to a sustainable model once USAID funding ends?  What challenges would external partners face to provide ABE services?  Do you know if there are other models of programs that transitioned to self-sustained activities? 6) Which AYP strategies/interventions do you feel have worked best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 7) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 82 8) Do you feel that AYP built the capacity of the teachers/facilitators and administrators?  How is the MoE supporting them? 9) Do you have contingency plans for emergency situations, to keep providing instruction for your learners? If so, what are they?  Did the learners use the Interactive Radio or mLearning tools to support their studies? Do they find them helpful? 10) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 11) Are physical contributions like the solar lights that USAID has made to the school sustainable? 12) Do you have recommendations for AYP going forward? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 83 Advancing Youth Project (AYP) Performance Evaluation KII with Current/Former ABE Learners 1. County: 2. Date: 3. Gender: M F 4. Age: 5. Mother Tongue: 6. Are you employed? If yes, Full time? Part time? Informally? Business? 7. Do you have dependents? If yes, how many? 8. How many times a month do you feel it is hard to provide for their basic needs (food shelter)? Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). 1) What is your educational background?  Are you currently taking the AYP classes? If yes which level are you in?  If you have already taken the AYP classes, what level did you complete? 2) What are your opinions of the AYP?  What is your opinion of the curriculum in your classes’ curriculum?  Are you satisfied with how the ABE program was managed and provided to you?  Are you satisfied with the facilitator/site administrator? 3) Which parts of the program have worked best for you to keep you engaged in the program, or to bring other learners to classes?  Did you use the Interactive Radio or mLearning tools with your studies? Did they help you? 4) What strategies could the program use to encourage learners to complete all three levels?  Are there differences for male and female learners? 5) How much have you improved your reading, math, and work readiness skills as a result of this program?  Do you think this is different for male and female learners? 6) Have you had any interactions with any local school/community groups (like youth clubs, ABE Committees, County Alliances) associated with this program?  If so, was this been beneficial to you?  Are there other services these groups could offer to make it a more successful learning experience? 84  Do you think this is different for male and female learners? 7) Have you participated in any trainings or activities to help you learn skills you could use to earn money through a job or individual activity?  If so, what were they?  If so, did you use these skills to earn money? If so, in what way?  If so, do you think the training matched the local job market or opportunities to earn money?  Do you think this is different for male and female learners? 8) Did you participate in an internship or on-the-job training through this program?  If yes, what was it?  If so, did you use this opportunity in some way to work or to earn money? If so, in what way?  If so, do you think the training matched the local job market or opportunities to earn money?  Do you think this is different for male and female learners? 9) Have you found a job after being part of the AYP?  What more could the project do to help youth find a job or start a business? 10) Have you participated in any village saving and loans activities (VSLA)?  If yes, what do you see as the positives and negatives of this experience? 11) How will AYP help you in the future?  What parts of the AYP are the most successful that should continue in the future after the project ends?  How can communities continue AYP activities after the project ends? Are their ways to raise funds for AYP activities at the community level? 12) Challenges & recommendations  What main challenges have you faced in the AYP?  What other main recommendations do you have for the AYP? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 85 Advancing Youth Project (AYP) Performance Evaluation Focus Group Discussion (FGD) with Current/Former ABE Learners [Respondents to fill out an attendance sheet that will include demographic questions] Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 60+ minutes. We want all of you to participate in this FGD and we will ask all of you to provide your opinions. We will take notes of our discussion (or record the conversation with their permission). 1) How many of you are currently taking the AYP classes? What level are you in? (show of hands) How many of you have already taken the AYP classes? What level did you complete? (show of hands) 2) Which parts of the program have worked best for you and kept you engaged in the program?  Did you use the Interactive Radio or mLearning tools with your studies? Did they help you? 3) How much have you improved your reading, math, and work readiness skills as a result of this program?  Do you think this is different for male and female learners? 4) Who has been involved with organizations like youth clubs, ABE Committees, County Alliances for this program? Was it useful? How could it be made better?  Do you think this is different for male and female learners? 5) Have you participated in any trainings or activities to help you learn skills you could use to earn money through a job or individual activity?  Do you think this is different for male and female learners? 6) Did you participate in an internship or on-the-job training through this program?  Do you think this is different for male and female learners? 7) How will AYP help you in the future? What other main recommendations do you have for the AYP? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 86 Advancing Youth Project (AYP) Performance Evaluation Focus Group Discussion (FGD) with Youth Clubs Leaders [Respondents to fill out an attendance sheet that will include demographic questions] Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). . 1) How many of you are currently taking the AYP classes? What level are you in? (show of hands) How many of you have already taken the AYP classes? What level did you complete? (show of hands) 2) What is your opinion of the curriculum in your classes – specify by Level if appropriate?  Are you satisfied with how the ABE program was managed and provided to you?  Are you satisfied with the facilitator/site administrator?  Did you use the Interactive Radio or mLearning tools with your studies? Did they help you? 3) What AYP strategies/interventions have worked best for mobilizing, recruiting and retaining you and other youth? 4) What AYP interventions were the most successful in helping to create positive outcomes for youth to complete all three levels of the ABE?  Do you think this is different for male and female learners? 5) Do you think that AYP activities are sustainable if USAID funding ends?  Which activities would continue?  What challenges would you expect to face if you had to transition to a self-sustained program?  Do you know if there are other models of programs that transitioned to self-sustained activities? Is this a realistic strategy? 6) To what extent have you and other youth improved their literacy, numeracy and work readiness skills as a result of AYP?  Do you think this is different for male and female learners? 7) How much do you interaction with other local school and community groups (ABE committees, County Alliances)?  Is this useful for you and the youth club overall? 87  What more do you think these groups can do to support your work? 8) Did you participate in livelihoods trainings/activities?  What did these consist of?  Do you think that the training addresses local market needs?  Do you think the training results in employment/business opportunities for youth?  Do you think this is different for male and female learners? 9) Did you participate in any internships or on the job training from AYP partners?  What did this consist of?  Do you think these opportunities address local market needs?  Do you think these opportunities result in employment or business opportunities for youth?  Do you think this is different for male and female learners? 10) Have you obtained any employment after being part of the AYP?  What more can be done by the AYP to help youth gain employment/start a business? 11) Have you participated in any village saving and loans activities (VSLA)?  What has been the result of your participation in the VSLA? 12) How will AYP help you in the future?  What parts of the AYP are the most successful that should continue in the future after the project ends?  How can communities continue AYP activities after the project ends? Are their ways to raise funds for AYP activities at the community level? 13) Challenges & recommendations  What main challenges have you faced in the AYP?  What other main recommendations do you have for the AYP? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 88 Advancing Youth Project (AYP) Performance Evaluation FGD with School/Community Partners (County Alliances, ABE Committees) [Respondents to fill out an attendance sheet that will include demographic questions] Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Advancing Youth Project (AYP), which was implemented by Education Development Center Inc. The purpose of this performance evaluation is to document lesson learned to date and to inform the design of future education programs of a similar nature. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the AYP. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with their permission). 1) What are your responsibilities within the AYP project? 2) Do you feel that your expectations and needs were met with AYP?  What expectations and policy needs still need to be met? 3) Are ABE program models flexible and realistic strategies to meet the program’s goals? 4) Do you feel that the AYP curricula are effective learning tools? Are they appropriate for the learning context of Liberia?  Do you think the MoE will be likely to the curriculum/materials in the future? If so, in what way? 5) To what extent do you think that the AYP model is sustainable if it relies only on local funding?  If yes, which elements do you think would continue?  If no, what would be required in order to sustain the activities that were started by AYP? What are the barriers to continuing this work? 6) Which AYP strategies/interventions work best for mobilizing, recruiting and retaining out of school youth?  Creating positive outcomes for youth to complete all three ABE levels?  Have these worked equally well for both male and female youth? 7) Do you feel that AYP is an effective tool to help learners improved their literacy, numeracy and work readiness skills?  Equally for men and women? 8) Do you feel that AYP built the capacity of the teachers/facilitators and administrators?  How is the MoE supporting them? 89 9) Do you feel that AYP is successful in linking youth to relevant and realistic livelihoods trainings/activities & employment/business?  If so, which interventions are most successful to create positive outcomes  Males and females benefitted equally from these trainings? 10) Do you have recommendations for AYP going forward? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 90 ANNEX IV: SOURCES OF INFORMATION Key Informant Interviews – Activity Management and Implementation Affiliated Group Type of Respondent USAID Former USAID/W Staff Current USAID/Liberia Education Office staff EDC Home & Central Office Project Director Chief of Party and Deputy Chief of Party for Technical Programming Learning Agenda Specialist Literacy and Numeracy Specialist Former COP Curriculum Specialist M&E Team Teaching and Learning Team Lead Institutional Capacity Development Team Lead MOE Central Office Minister Proper Deputy Minister for Policy, Planning and Research Assistant Minister for Basic and Secondary Education ABE Director County Education Officer – Grand Bassa County District Education Officer – Nimba County District Education Officer – Montserrado County MOYS Minister Proper AYP Staff Team Leader – Montserrado County Team Leader – Nimba County Team Leader – Grand Bassa County ABE Coordinator – Montserrado County ABE Coordinator – Grand Bassa County M&E Specialist – Grand Bassa County YDLCs – Montserrado County YDLCs – Nimba County YDLCs – Grand Bassa County Other Partners VTC – LOIC Grand Bassa VTC – LOIC Montserrado County Alliance – Nimba County Alliance – Grand Bassa YMCA WorldORT NAEAL LIYONET European Union 91 Private Sector Partner – Arcelor Mittal 106 Garage Key Informant Interviews - School Sites Affiliated Group Type of Respondent Gray D Allison Administrator Facilitator Kortu Town Administrator & Facilitator Sanniquellie Administrator & Facilitator Youth Club President Gehwee Youth Club President New Yekepa Administrator Tondin Facilitator Classroom Observation Little Bassa Administrator Learner KII Grow Your Business Facilitator Benson River Administrator Cotton Tree Administrator Facilitator #1 Facilitator #2 Facilitator #3 St. Paul’s Bridge Administrator/Facilitator Lookingtown Central Administrator Other Beneficiaries AYP Graduate LOIC Graduates – Grand Bassa Graduates – Montserrado Focus Group Discussions Affiliated Group Type of Respondents # of Respondents Grey D Allison Learners 12 Kortu Town Learners 19 Sanniquellie ABE Committee 6 Learners 12 Gehwee Administrator, Facilitators, & Learners 27 New Yekepa Facilitators & Learners 22 Tondin ABE Committee 6 Learners 16 Lower ABE Committee 2 92 Harlandsville Learners 10 Little Bassa Facilitators & Learners 19 Grow Your Business Learners 7 Benson River Facilitators & ABE Committee 10 Learners 17 93 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 94 95 96 ANNEX VI: AYP PROJECT OVERVIEW The Advancing Youth Project (AYP) provides alternative basic education (ABE) programming, including literacy and numeracy, work readiness, and life skill, training, to Liberian youth who never received any formal schooling and others who dropped out of school in early primary grades. The project is implemented with Basic Education funding from the USAID/Liberia Education Office, with a portion of funding coming from the Economic Growth office to support vocational skills training, and a portion from the Health Office to support healthy lifestyles curriculum. The basic education learning component of AYP supports out of school youth to improve their literacy, numeracy, work readiness and life skills through the provision of ABE. The program is open to all youth within the age range of 13-35 who reside in or near the communities where the AYP schools are located.9 The project data shows that since its inception more than 22,000 youth have enrolled in the ABE classes, the average age of the learner is 26, the majority of learners are women (60-70 percent); however there is also a high dropout rate: out of the 22,256 enrolled, only 3,103 completed the level. Table 1: AYP Learners Enrolled, Completed, and Employed; Administrators and Facilitators Trained Indicator Aggregate # % Male % Female Learners Enrolled 22,256 42 58 Learners Completed the Level for which Enrolled 3,103 36 64 Learners Employed 136 30 70 Administrators Trained 289 89 11 Facilitators Trained 983 83 17 Curriculum AYP comprises a three-level ABE curriculum, which is directly aligned with the MoE public schools curriculum for primary grades 1-6. The curriculum10 was developed by technical specialists from EDC and AYP, and was overseen by a Technical Working Group (TWG) consisting of pedagogy experts and officials from MoE, UN organizations and international NGOs in Liberia. After incorporating the comments of the TWG the curriculum was implemented in AYP schools. Based on feedback received after its implementation from learners, facilitators, administrators and MoE officials, the Level 1 and 2 curriculum was revised over the course of the project. AYP does not have sufficient time to revise the curriculum for Level 3 before the project’s conclusion, however, it will provide MoE all the pertinent information to do so in the future. Table 3 details the three levels, what subjects are covered, and their equivalencies in the conventional school system. 9 In some schools learners older than 35 also attend the classes and are allowed to stay. 10 The curriculum also builds upon the one used in the USAID’s Core Education Skills for Liberian Youth (CESLY) project, an accelerated learning program (ALP) which was also implemented by EDC in Liberia from 2009-11. 97 Table 2: AYP Classes and Subjects by Level Level Subjects Classes Equivalency Level 1 Literacy, Numeracy, Work Readiness and Life Skills11 --3 ½ hours per class held four days a week12 --2 semesters spread over 9 to 10 months --Classes usually start in September --Schedule generally aligned with the MoE academic calendar Learners are at a 2nd grade equivalency at successful completion. Level 2 Literacy for Work, Numeracy and Life Skills Same as above Learners are at a 4th grade equivalency at successful completion. Level 3 --English, Math, Social Studies and Science --The ‘Grow Your Business’ course was added in 18 AYP schools Same as above Learners are at a 6th grade equivalency at successful completion. Learners are assigned to one of the three levels of the ABE classes based on initial placement tests, which are administered by AYP before the classes start. At the end of each level the learners take a completion test to move on the next level (if they are in Levels 1 or 2) or to complete the ABE program (i.e. Level 3). If a student does not pass the completion test, he/she is encouraged to repeat the level. Those youth who successfully complete Level 3 obtain a certificate from AYP.13 All the subjects (for a level) are taught in each class, however, more time is spent on literacy and numeracy than on the other subjects. AYP schools are also provided with classroom facilitators’ guides and learners workbooks for all the subjects in the three levels. The ABE curriculum is also offered with an interactive audio instruction (IAI) support for Level 1. Initially this was only for literacy courses, but now there are audio lessons for all the four subjects. AYP provided each school a radio with the taped audio programs (usually 15 minutes in length) and facilitators guides for IAI. Facilitator and administrators were also trained in using and teaching the IAI lessons. The curriculum also has a mLearning component for Level 2 (Literacy for Work and Numeracy) and Level 3 (English Language and Mathematics). Questions are sent, over mobile phones (with costs paid by AYP) to facilitators to discuss in class; and to learners to answer the topics covered in the classes. Some Level 2 learners who had cell phones were trained on the mLearning component of AYP, and they then trained other learners in their levels. 11 Work Readiness and Life Skills are not offered as courses by MOE in primary schools, but topics under these subjects are covered in the government curriculum. 12 AYP schools closed from August 2014 to February 2015 due to the Ebola crisis, and when they resumed AYP decided to offer classes 4 days a week (instead of 3) to make up for the lost time. 13 The MOE does not provide any equivalent certification for AYP graduates of Level 3. 98 AYP is also using two tools to measure learners’ gains in literacy and numeracy: The Out-of-school Literacy Assessment (OLA), which was specifically created for older youth and young adults who have minimal to no literacy skills; and the Early Grade Math Assessment (EGMA) an early grade numeracy tool to assess learners’ fundamental knowledge of numeracy and mathematics. An EGMA baseline was administered to an AYP cohort in 2012 and an endline to the same cohort in 2013. An OLA assessment was also done with two AYP groups and a control group (in 2012/2013). Further, an OLA baseline was administered to AYP cohorts entering Level 1 and 2 in June/July 2015; and an EGMA baseline was also administered to another AYP cohort entering Level 3 during the same time period. The endlines of these two assessments will be administered to the same AYP cohorts (completing the three levels) in June 2016. Administrators and Classroom Facilitators Each AYP school has one administrator who manages the program at the school and classroom facilitators who teach the three levels.14 Generally, the administrator is the principal and the facilitators are teachers at the MoE day school, which serves as the AYP school site. After choosing the school sites, principals and teachers were asked to apply to join AYP. For facilitators, their school performances and the subjects they teach (in the day school) were also considered. All administrators and facilitators were provided an initial 10 day training and also had to complete pre-tests and post-tests. Based on the results of the pre and post tests and their performance during the trainings, successful candidates were selected; and in the case of facilitators assigned to teach one of the three levels, and the appropriate subjects for that level. All the facilitators and administrators are also paid a monthly stipend for the duration of the ABE classes. Levels 1 and 3 each have two facilitators,15 with each facilitator teaching two subjects; while Level 2 has one facilitator who teaches all three subjects. The administrator is also supposed to teach any subject in the three levels as the need arises. Facilitators and administrators have also received follow up trainings to improve their capacity to teach the ABE curriculum. AYP has also formed a partnership with Stella Maris Polytechnic University to offer a pilot credit course (with a course curriculum, assessment and evaluation tools and a corresponding resource book) for teaching ABE. Some AYP facilitators have also participated in trainings offered by Stella Maris on ABE. Groups Associated with AYP Schools The learners in each school form youth clubs which have an elected leadership structure including a president, vice-president, secretary and treasurer. Youth clubs play a role developing networks among learners, conduct other activities such as community clean-up and aid in recruitment and retention of learners for the ABE classes. Further, youth clubs mobilize their members to set up small gardens on land provided by the community (where they grow crops such as cassava, peppers etc. to sell in local markets); and members who have gotten skills trainings (for example in soap making) are also active in producing and selling the soap. In addition youth club leaders have also received trainings on leadership. 14 Some of the AYP schools do not offer a level based on a lack of learners enrolling in the level. 15 The Grow Your Business subject for Level 3 has 2 different facilitators (one for agriculture and one for business) 99 Youth summits are held at the national and county levels where two members from each site (usually youth club leaders or other motivated learners) are invited to attend. In these summits the participants present the activities they carried out in their communities and discuss their ABE learning and livelihood trainings, as well as their successes and challenges. These summits further provided an opportunity for youth to build their leadership skills. Each AYP school has an ABE committee which includes community chiefs, youth and women leaders and other prominent members of the community. The ABE committee also has an elected leadership structure including a chairperson, co-chairperson, secretary, recruitment coordinator and advisor. The school administrator and youth club president also serve in the ABE committee. The ABE committee oversees the following: that the ABE classes are being held on time; the administrators and facilitator are teaching regularly; the quality of the teaching of the facilitators; plays an active role in recruitment and retention of learners; advises the youth clubs and the learners; oversees the materials provided by AYP to the schools; and plans for the future of ABE in the community. A county alliance also exists in each of the counties where AYP is operational; and consists of county level officials from the MoE and MoYS, NGOs and CBOs leaders, priests, business owners and other leading community members. The main role of the county alliances is to work for the improvement of youth’s lives. Other ABE Partners AYP has two other partners in the provision of ABE to Liberian youth. LIYONET is using the AYP curriculum in 15 school sites in Nimba County, with facilitators who are also MoE school teachers. NAEAL is offering the AYP Level 1 curriculum in 20 schools in two counties, however, they are using their own facilitators. NAEAL also has its own ABE curriculum that it offers to adults across the county. AYP County Setup In each AYP county the project staff consists of the following,  Team Leader: who is responsible for managing the whole AYP program in the county  ABE Coordinator: who is responsible for overseeing the learning component of the AYP including overseeing the facilitators and administrators and their training  Monitoring and Evaluation Officer: who is responsible for overseeing the data collection for the indicators of the project  Three Youth Development and Livelihoods Coordinators (YDLCs): who are YMCA employees and oversee the livelihoods component of the project  Other Staff: finance officer, office assistants, interns, drivers, security etc. Livelihood Skills Training Through result 3, the project placed more emphasis on improving access for learners to acquire the relevant skills and opportunities needed to improve their livelihoods. The learners have varying low skill backgrounds with many engaged in farming (in rural areas) and others in fishing, menial jobs, small businesses, construction work (in urban areas) etc., while some are unemployed. Some of the learners (based on their regular attendance in classes, leaderships in youth clubs) have been provided short-term skills trainings in soap making, baking, bee keeping etc. While a smaller subset of youth (usually in Level 2 and 3) have been provided long-term (16 months) vocational trainings (at LOIC) in mechanics, electric, 100 masonry, carpentry, welding etc. Short term (3 months) months internships at garages, tailoring shops etc. are also provided to some learners in certain counties. Considering the age range of the learners, it is vividly clear that most of them will remain in the informal sector and AYP had therefore placed emphasis on providing youth with the skills and knowledge necessary to improve their own businesses. Notwithstanding, the project made it mandatory for learners to concentrate and understand the linkages between the education they have received in ABE classes and its application outside the classroom. Through this result, AYP has provided over one million person hours of training in workforce development through a combination of classroom teaching of work readiness skills, business skills and agriculture-related skills, as well as short-term skills training in areas which learners have master fairly quickly. They have recently introduced these skills without requiring significant capital investment in tools and materials, and are currently marketable within their communities. 101 ANNEX VII: OLA AND EGMA SUBTEST RESULTS OLA SUBTEST RESULTS Subtest Literacy Skills Findings Letter Naming The minimum prerequisite for early reading development in English is usually 26/26 letters Of the total respondents over half (52.7%) could read 19-25 letters. Approximately 11% could read either 7-12 or 13-18 letters, while 11.2% could not read any letters at all. More males were able to read all 26 letters and females were more likely to read less than 13 letters correct (38.8%) as compared to males (21.5%). Letter Sounds Generally, beginning readers need to be able to identify letter sounds as well as letters names to be able to ready quickly and accurately. Intervention Group #2 (IAI) learners made greater gains than Intervention Group #1 learners. The comparison group learners’ scores declined slightly at the endline. However, scores were extremely low, with only 0.9% of the respondents able to sound out all or almost all letters correctly. Real Life Reading Questions Items are designed to give respondents an opportunity to demonstrate their literacy skills in real-life contexts, i.e. value of Liberian money, labels on food items, and signs on buildings. Level 1 learners answered correctly between 1 in 3 and 1 in 4 real life items. Learners were able to answer the locator items (finding the word rice on a bag), but had more difficulty with inferential items where they had to apply the information (medicine bottle). Word reading (100 words) Rapid, accurate word recognition is critical for successful reading and comprehension. 2 out of 5 respondents were not able to read any words, and nearly 95% read 20 words or less. Only Intervention Group #2 (IAI) did not show a slight decline in scores at the endline. Story 1-Oral passage reading (33 words) This Grade Equivalency (GE) 1 story is similar in content and readability to Level 1 curriculum materials and measures reading of connected text. Only one in five learners were able to read all or nearly all words in the story at the endline. Disaggregation of the results by locale shows that urban Intervention Group #2 (IAI) participants more than doubled their results at the endline, compared to the baseline. The comparison group’s fluency rate did not change. Story 1-Reading comprehension Only respondents who were able to read ten or more words in the passage did this. Level 1 learners showed a gain in average percentage of correct answers at endline. The endline average was still low, just over 20%. Story 2-Oral passage reading (85 words) This story is similar in content and readability to Level 1 curriculum materials. Only respondents who were able to answer two comprehension questions on Story 1 continued on to Story 2. Level 1 learner groups were able to read approximately 40% of the story correctly at endline. Intervention Group #1 showed a gain of 15.8% over baseline and Group #2 showed a 18.5% gain. The increase for the Comparison Group was 7.26%. Story 2-Reading comprehension Those learners that could answer at least two Story 1 comprehension questions attempted this story. At the endline, 33.9% of learners attempted Story 2. Learners from Intervention Group #2 showed significant improvement between the baseline and the endline, while other learner groups did not. Silent reading comprehension Those learners that could answer at least two Story 2 comprehension questions attempted this passage. Less than 2% of Level 1 youth attempted the silent reading at the endline. Over half of Level 2 endline learners did not answer a single 102 comprehension question correctly, and no one answered all five comprehension questions correctly. EGMA SUBTEST RESULTS Subtest Numeracy Skill Findings Number Identification Measures ability to recognize numbers quickly. The mean gain score for correct numbers identified was 5.9%. On the posttest, over half of the respondents were able to identify 8 or more of the 20 numbers correctly. Number Discrimination Measures ability to compare quantities expressed numerically quickly. The mean gain score was very low at 5.9%. On the posttest, only a quarter of the respondents were able to complete 8 to 20 problems correctly. Missing Number Measures ability to recognize a missing number in a pattern of three numbers. The mean gain score was 2.8%. On the posttest less than7% were able to answer at least 4 questions correctly. Addition Part 1 measures ability to add single digit whole numbers quickly. Part 2 is addition double digit whole numbers and only attempted by those that answer 1+ on part 1. On part 1, the mean gain score for percent correct was 11%. Respondents were able to complete nearly half of the ten questions correctly. The gain score for part 2 was 7.1%. 41% of those who attempted part 2 did not answer any two￾digit problems correctly, and 10% answered 8 or more correctly. Subtraction Part 1 measures ability to subtract single digit whole numbers quickly. Part 2 is subtraction of double digit whole numbers and is only attempted by those that can answer 1+ on part 1. The mean gain score for Part 1 (percent correct) was 10.1%. At posttest 20% of respondents could not answer any questions correctly, while almost 10% could answer 8 to 10 questions correctly. The gain score for Part 2 was 7.1%. Of those that attempted Part 2, only 58.2% answered 1 or more problems correctly (10% or more). Multiplication Of six multiplication questions, four problems include multiplication of single digit whole numbers, and 2 problems include multiplication of one double digit whole number by a single digit whole number. The mean gain score was 3.3%, but this was statistically significant. The mean on the posttest was 13.2%. However, 47% did not answer any (zero score) post-test multiplication questions correctly. Division Of six division questions, four problems include division of single digit whole numbers, and 2 problems include division of one double digit whole number by a single digit whole number. The gain was less than 1%, and was not statistically significant. The mean percentage answered correctly at the posttest was 7.9% and 64.3% of respondents did not answer any questions correctly. Shape Recognition Measures ability to identify circles, squares, triangles, or rectangles among other shapes and objects. At the posttest the respondents were able to identify half of the shapes, with a gain score of 6.06%, which was statistically significant. Only 7.7% were not able to identify any shapes on the posttest. The majority (70%) were able to identify 41-100% of the shapes. 103 ANNEX VIII: OLA/EGMA BASELINE STUDY 2015/16 A baseline was done for OLA and EGMA in June/July 2015 with the endline expected to be conducted in June 2016. The OLA baseline was administered to a random sample of 642 youth in five counties in Liberia who are enrolled in either Level 2 or Level 3. The EGMA was only administered to Level 3 learners (300 learners) to assess baseline numeracy skills. The baseline OLA assessment showed that,  Youth in Level 3 showed significantly (p<.001) better literacy skills than their counterparts in Level 2, with the exception of the Letter Sounds subtest.  The majority of Level 3 youth have moved beyond the beginning stages of literacy development, but have not developed sufficient reading comprehension skills, particularly silent reading comprehension, that would allow them to “read to learn.”  Males in Level 2 performed significantly better than females on all the OLA subtests with the exception of Letter Sounds.  In Level 3, the gender gap in literacy performance was less pronounced with males and females performing relatively similarly.  Level 2 youth in urban areas performed significantly better than their rural counterparts in five out of eight OLA subtests, after controlling for the sex of the learner.  Disaggregation by locale for Level 3 youth showed little difference in average OLA subtest scores for rural and urban youth. The baseline EGMA assessment showed that,  Level 3 youth performed the best in the most elementary and procedural of subtests— number identification and addition. By contrast, the subtests with the lowest scores, in which youth particularly struggled, were ones focused on more advanced mathematical operations and conceptual understanding — missing numbers, multiplication and division.  Males performed better than females on all EGMA subtests. Differences between males and females were statistically significant on all subtests with the exception of Missing Numbers (pattern identification) and Subtraction.  Overall rural youth performed slightly better than their urban counterparts on eight out of ten EGMA subtests, however, the differences in rural/urban performance were not statistically significant. The endline results of both OLA and EGMA will demonstrate the improvement in the learners’ literacy and numeracy skills. However, as the endlines will be administered after the timeframe of this evaluation, those results cannot be discussed in this report. 104 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523