September, 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by PRIN International Consultancy and Research Services PLC(prin.antil@yahoo.com; tilayek@gmail.com). Final Performance Evaluation of the School￾Community Partnership Serving Orphan and Vulnerable Children Affected by HIV/AIDS (SCOPSO) Project FINAL PERFORMANCE EVALUATION OF THE SCHOOL-COMMUNITY PARTNERSHIP SERVING ORPHAN AND VULNERABLE CHILDREN AFFECTED BY HIV/AIDS (SCOPSO) PROJECT SUBMITTED TO: USAID/Ethiopia SUBMITTED BY: PRIN International Consultancy and Research Services PLC SEPTEMBER, 2013 DISCLAIMER This publication was produced at the request of the United States Agency for International Development. It was prepared independently by PRIN International Consultancy and Research Services PLC (prin.antil@yahoo.com; tilayek@gmail.com) under SOL-663-13-000005. The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii CONTENTS Acknowledgements .................................................................................................................................... v Acronyms....................................................................................................................................................vi Executive Summary..................................................................................................................................vii 1. Introduction ............................................................................................................................................. 1 1.1 Background.............................................................................................................................................................. 1 1.2 Intervention Model ................................................................................................................................................ 2 1.3 Purpose of the performance Evaluation ........................................................................................................... 2 1.4 Evaluation Questions............................................................................................................................................. 3 2.Design and Methods of the Evaluation.................................................................................................. 4 3. Findings ................................................................................................................................................... 6 3.1 PROJECT DESIGN AND RELEVANCE ........................................................................................................... 6 3.1.1Project design ................................................................................................................................ 6 3.1.2 Relevance to community needs ................................................................................................... 7 3.1.3 Relevance to the government strategy ........................................................................................ 9 3.1.4 Participation of stakeholders...................................................................................................... 10 3.2 EFFICIENCY.......................................................................................................................................................... 10 3.2.1 Input ........................................................................................................................................... 10 3.2.2 Efficiency of the Project ............................................................................................................. 11 3.2.3 Costs by Service Area ................................................................................................................ 12 3.3 PROJECT EFFECTIVENESS ............................................................................................................................... 13 3.3.1 Education Support ...................................................................................................................... 14 3.3.2 Psychosocial Support ................................................................................................................. 15 3.3.3 Food and Nutrition .................................................................................................................... 17 3.3.4 Healthcare .................................................................................................................................. 18 3.3.5 Shelter and Care ........................................................................................................................ 20 3.3.6 Economic Strengthening ............................................................................................................. 21 3.3.7 The situation of non-intervention schools as indicators of the effectiveness of the project .... 24 3.3.8 Capacity Building ........................................................................................................................ 25 3.3.9 Clients’ Impression about the Project ....................................................................................... 27 3.4 SYSTEM AND STRUCTURE............................................................................................................................. 28 3.4.1 Beneficiary Selection Mechanism ............................................................................................... 29 3.4.2 Project Coordination and Implementation Structure................................................................ 29 3.4.3 Community Mobilization ............................................................................................................ 31 3.4.4 Monitoring and Evaluation ......................................................................................................... 32 3.5 EMERGING IMPACTS ........................................................................................................................................ 33 3.6 OPPORTUNITIES AND CHALLENGES........................................................................................................ 34 3.7 SUSTAINABILITY OF THE PROJECT OUTCOMES.................................................................................. 35 4. Conclusions...........................................................................................................................................36 5. Lessons Learned...................................................................................................................................37 6. Recommendations ................................................................................................................................38 References.................................................................................................................................................41 Annexes .....................................................................................................................................................42 ANNEX I: SCOPE OF WORK................................................................................................................................ 42 ANNEX II: DATA COLLECTION INSTRUMENTS .......................................................................................... 56 iii ANNEX III: DOCUMENTS REVIEWED............................................................................................................... 74 ANNEX IV: DISCLOSURE OF ANY CONFLICT OF INTEREST.................................................................. 75 ANNEX V: ACHIEVEMENT AGAINST SELECTED INDICATORS.............................................................. 82 ANNEX VI: COVERAGE OF THE PROJECT AGAINST CORE SERVICES................................................ 83 ANNEX VII: LIST OF INSTITUIONS & INDIVIDUALS CONTACTED ...................................................... 84 ANNEX VIII: SAMPLE SCHOOLS BY REGION & INTERVENTION STATUS.......................................... 88 iv ACKNOWLEDGEMENTS The Evaluation Team is grateful to many persons who helped to facilitate and expedite the work of the Team in this Final Performance Evaluation of the School-Community Partnership Serving Orphan and Vulnerable Children Affected by HIV/AIDS (SCOPSO) Project. First and foremost, we would like to thank Ato Befekadu Gebretsadik, USAID Project Management Specialist, for arranging the evaluation and for his guidance and assistance in making this evaluation a reality. Along with other key USAID staff, he prepared the scope of work for the evaluation, which gave us clear direction for the task of evaluation. We are grateful to AtoTahir Gero (COP for SCOPSO of World learning Ethiopia); Ato Aragaw Biru (DCOP for SCOPSO of World learning Ethiopia); and Ato Worku Ambelu (Monitoring and Evaluation Specialist for SCOPSO of World learning Ethiopia) for their support and assistance during the assignment. They had selected sample project supported schools and arranged for many of the key stakeholder interviews. Our thanks also go to Area Coordinators and project officers for their excellent facilitation during the field work. We would also like to thank other members of the USAID/Ethiopia Mission for their assistance and advice on the Evaluation. We also appreciate the attention paid by the entire Mission management team to the evaluators’ debriefing and providing them with valuable and constructive comments. The evaluation team benefited greatly from the debriefing and meetings with the World Learning Ethiopia project staff in order to better understand the strengths and constraints of the project. Finally, we would also like to thank the representatives of the various SCOPSO supported school communities that have worked with SCOPSO to achieve project objectives. The list is too long to mention here but these school communities and their comments are reflected throughout this report. They provided essential information that allowed us to better understand the challenges of their schools, the relationships between them and the SCOPSO staff and potential for achieving greater impact in the future. Thank you, The SCOPSO Final Performance Evaluation Team TilayeKassahun, PhD (Team Leader) v ACRONYMS AIDS Acquired Immuno-Deficiency Syndrome CBO Community Based Organization CSI Child Status Index FGD Focus Group Discussion GAC Girls’ Advisory Committee GOE Government of Ethiopia HIV Human Immuno Deficiency Virus KETB Kebele Education and Training Board MOE Ministry of Education MOH Ministry of Health NGO Non-Governmental Organization. OVC Orphaned and Vulnerable Children PEPFAR The US President Emergency Plan for AIDS Relief PTA Parent Teacher Association RSEB Regional State Education Bureau SCG School Core Group SIA School Incentive Awards SOW Statement of Work SSDG Standard Service Delivery Guideline TWG Technical Working Group USAID U.S. Agency for International Development USAID/E U.S. Agency for International Development- Ethiopia VSLA Village Saving and Loan Association WEO Woreda(district) Education Office WL-E World Learning-Ethiopia vi EXECUTIVE SUMMARY In January 2013 USAID/Ethiopia requested a Final Performance Evaluation of the School-Community Partnership Serving Orphan and Vulnerable Children Affected by HIV/AIDS (SCOPSO) project implemented by World Learning - Ethiopia under its Solicitation Number: SOL-663-13-000005. PURPOSE OF THE EVALUATION The overriding purpose of this performance evaluation is to gain an independent view of the overall performance of the project in order to help USAID/E Basic Education Office, Ministry of Education, Regional Education Bureaus, schools, communities and other stakeholders with regard to the overall achievement of the project’s intended results. Based on the implementation results of SCOPSO, USAID/E is also interested in learning more about what worked, what did not work and challenges in terms of the education of orphan and vulnerable children at schools. The evaluation will also help to learn what has been attained and the challenges faced during implementation for future similar project design and implementation in line with the Mission Country Development Cooperation Strategy (CDCS), Development Objective 3 -Improved Learning Outcomes; IR 3 - Increase achievement in basic education particularly in early grade reading. EVALUATION QUESTIONS The evaluation intended to answer the following four key questions: 1. How effective is the project in achieving its set objectives and anticipated results? 2. How is the project’s approach and methodology designed to achieve the project objective and results? 3. What are the levels of satisfaction of the Ministry of Education, Regional Education Bureaus, Woreda Education Offices, schools, communities and beneficiary OVCs and their parents/ guardians with the project? And why? 4. What are the lessons learned from the project? What are challenges encountered and measures taken? PROJECT BACKGROUND The School Community Partnership Serving Orphan and Vulnerable Children in Primary Schools (SCOPSO) initiative, which is funded by USAID through PEPFAR and implemented by World Learning￾Ethiopia, started in August 2009 and will end in 2014. The project aimed in part to strengthen the ability of schools and communities to participate actively in the design, implementation and management of OVC support activities at schools in sustainable way. The overall objective of the project was to build the capacity of 400 primary schools to serve as focal points for OVC care and support to at least 52,000 HIV affected or infected OVC leading to increased enrolment, retention and academic performance. In order to attain the objective of the project, standardized core services including education, health care, food and nutrition, psychosocial and life skills trainings, shelter and care and economic strengthening focusing on income generating activities were provided to OVC at schools. The School Core Groups (SCG) played a major role in mobilizing the wider community and key influential actors such as religious leaders, community elders and various Community-Based Organizations (CBO). vii In June 2013 USAID/E contracted with PRIN International to conduct an external evaluation “to ascertain the overall performance of the project and to learn achievements and challenges faced during implementation of the project.” DESIGN AND METHODS OF THE EVALUATION This evaluation, undertaken by PRIN International with a team of four consultants, took place in the months of June and July 2013. The evaluation is mainly qualitative in nature; and in order to obtain a reasonable picture of the strengths and weaknesses of project implementation, the team leader and the project manager in consultation with the Program Management Specialist from USAID/E and Senior Management members from WL-E had selected a representative sample of 41 SCOPSO intervention schools and 6 non-intervention schools in five regions and one city administration in Ethiopia based on purposive sampling method. All in all,1,005 people drawn from various stakeholders (OVC, guardians, SCGs, teachers, PTAs, KETB, WEOs, REBs, project coordinators, and other implementing partners) had participated in interviews and focus group discussions. Further, desk reviews have been made to capture relevant facts regarding the performance of the project so as to corroborate the data generated from primary sources.The teams provided a debriefing to both USAID Mission management and World Learning Ethiopia key staffwhose inputs have been duly incorporated into the report.. Limitation of the Study: Due to time constraint the size of sample schools taken for the study is small (nearly 10%) compared to the total population. However, all possible effort has been made to get adequate information about the project by using multiple tools to collect data from multiple sources. Critical review of reports has been used to get a good picture of the overall national situation of the project. FINDINGS AND CONCLUSIONS SCOPSO implemented its interventions at three levels: child, household and community. At the child level, SCOPSO primarily provided direct support on essential services – education, psychosocial, health, food and nutrition, shelter and care, and economic strengthening supports. Of all the supports, the education support has a leading role in determining the schooling of a child. The educational material support and food support were the two key interventions along with medical care to achieve improved retention, performance and enrolment. Due to limited budget (and restriction on direct food assistance) as well as limited resources made available from the local stakeholders, the food support could only address the needs of only less than20% of the beneficiary OVC. Besides, the engagement of the project on shelter and care was found to be highly successful in accomplishing more than what was planned but the scope of the service related to shelter was found to be very limited. At the household level, SCOPSO developed appropriate income-generating activities for OVC and their guardians. The economic strengthening support was carried out to ensure sustainability of care on the part of beneficiaries. However, only less than 22% OVC households were served by the IGA support. At community level, SCOPSO strengthened community capacity to support OVC and their guardians mainly through building the capacity of key stakeholders including Woreda HAPCO leaders, zone and Woreda education officials, healthcare providers, KETB and PTA representatives, CBOs and religious leaders, and Women’s Affairs Officials by facilitating training to identify the needs of vulnerable children and deliver appropriate services to OVC and their guardians. The project also provided training, coordination and material support in order to improve the coordination and management of the program. Hence, schools, community representatives and all other part-takers were capacitated to plan a 8 project, and manage implementation of the plan as well as to monitor and evaluate progresses. In so doing, WL-E had succeeded in taking the issue of OVC to community members, schools, and government officials at various levels so as to show their commitment towards improving the well-being and academic performance of the OVC. At the school level, SCOPSO included school incentive awards to initiate school-based income generating activities to finance programs for orphaned and vulnerable children, and provision of assets as per the preference of OVC households as well as through careful analysis of the contexts to improve their lives. Moreover, from the evaluation it has been learned that the project has been highly appreciated by almost all stakeholders who participated in the assessment. In particular, education officials at all levels had appreciated the project in achieving almost all set targets by involving relevant stakeholders at all levels. Comprehensiveness of the project, volunteerism and commitment of school community, sense of ownership on the part of the community, and notable commitment of WL-E project staff were identified to be among the main reasons for the positive impression the stakeholders developed about the project. Given this, it is possible to claim that the project is very effective in addressing its intermediate results. CHALLENGES Limited available budget especially for IGA and food support has been noted as the hindrance for full satisfaction of beneficiaries. Some schools are working with limited number of teachers and teachers are overstretched with regular school chores. Besides, the discontinuities of service provisions for OVC on non-school days/seasons were reported to be other major drawbacks connected to the project. Moreover, turnover among the trained staff in the intervention schools happened to be another major challenge for the project. KEY RECOMMENDATIONS Mirroring the findings and results of SCOPSO to date; including satisfaction at individual OVC, school and community levels; the evaluation team recommended that the program need to be strengthened and scaled up. However, some points need to be considered in the future intervention for management purpose. The evaluation team has suggested the possible measures to be taken against the key actor as follows: I. The Government (MoE, RSEB, WEO) 1.1. As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by encouraging and developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, both the federal and regional governments have to encourage those NGOs that are willing to work on OVC-related interventions to learn from such a good practice and devise a strategy to leverage the local resources by involving relevant stakeholders. 1.2. As it has been reported in the findings section, the success of the SCOPSO is partly attributed to the concerted efforts of school communities (viz., teachers, SCGs, PTAs, KETB, etc.) and other local partners' staff who provided voluntary service. In light of this, the evaluators recommend that the Government of Ethiopia (GoE) has to promote the idea of voluntarism and create a culture of voluntary service to support OVC in educational settings/schools. As 9 part of such a step, there should be negotiation with the WEO as well as the schools to reduce workload of teachers who commit themselves for such voluntary support. Such voluntary support could extend to charity clubs so that older members of the club support the younger OVC after proper training. This has added advantage of teaching compassion and charitable gift. 1.3. In order to ensure the continuity of the student learning and achievement, SCOPSO-type intervention needs to be introduced at secondary school level, particularly up to Grade 10. This helps those children who do not have any kind of support to complete their general secondary schooling and enhance their likelihood of gainful employment in the future. 1.4. One of the SCOPSO's major activities that fall under educational support intervention is provision of tutorial support for the target school children. In this regard, the SCGs had strongly recommend that SCOPSO’s intervention should also be closely aligned with Early Grade Reading and Writing Project so as to address the learning difficulties/needs of OVC as they are less likely to get the necessary home-and school-based remedial support. To facilitate this, RSEBs in consultation with other relevant partners, have to devise appropriate and feasible strategy in the shortest possible time. 1.5. In order to ensure sustainability of the project's outcomes, MoE and RSEBs need to consider earmarking school grants for the schools proportionate to the number of OVC they enrol. II. USAID 2.1. As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, USAID needs to encourage its implementing partners to plan for interventions which include strategies to leverage local resources by involving relevant stakeholders. 2.2. The SCOPSO project is a multi-dimensional intervention that requires the involvement of actors drawn from various sectors, such as health, agriculture, trade and industry, cooperatives, and so forth. However, the connection of the project to line sectors was found to be not as strong as expected. As a result, the children failed to get supportive services they had to from such public sectors (e.g. health). This could have been resolved had the project closely worked with those line offices at management level. Therefore, USAID need to encourage its implementing partners to form strong management relations in any similar project interventions in a way that there will be strong inter-sectoral integration. 2.3. In order to ensure the continuity of the student learning and achievement, SCOPSO-type intervention needs to be introduced at secondary level, particularly up to Grade 10. This helps those children who do not have any kind of support to complete their general secondary 10 schooling and enhance their likelihood of gainful employment in the future. Therefore, USAID needs to allocate resources to promote SCOPSO-type intervention at secondary schools. 2.4. The SCGs had strongly recommend that SCOPSO’s intervention should be closely aligned with Early Grade Reading and Writing Project so as to address the learning difficulties/needs of OVC as the OVC are less likely to get the necessary home-and school-based remedial support. USAID can promote this by connecting SCOPSO-type interventions to its READ Ethiopia projects. III. World Learning 3.1. As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, WL should learn from such a practice and devise a strategy to leverage the local resources by involving relevant stakeholders. 3.2. The SCOPSO project is a multi-dimensional intervention that requires the involvement of actors drawn from various sectors. However, the connection of the project to line sectors was found to be not as strong as expected. As a result, the children failed to get supportive services they had to from such public sectors (e.g. health). This could have been resolved had the project closely worked with those line offices at management level. Therefore, WL needs to form strong management relations in any similar project interventions in a way that there will be strong inter-sectoral integration. 3.3. There is a need to check the phase out strategy not to make that pull out disruptive. The remaining time of the project should be devoted to safe transition of the support from WL-E to Communities so that the communities as well as the schools can make sure that the children continue their education after the phase-out of the project. 3.4. Now that the OVC are in school. The project is also approaching its phase-out stage. It is therefore commendable that WL-E considers conducting follow up or tracer studies after phase out of the project. The output of such tracer studies should be able to inform on the proportion of children who could continue to the secondary level of education and those who continued to work their business that was initiated by SCOPSO. 3.5. The project approach for reimbursing medical expenses somehow dissatisfies the beneficiaries because they do not have enough to pay in advance. Therefore, SCOPSO has to devise appropriate and feasible strategy that facilitates easy access to the needed medical service for the beneficiaries. For example, SCOPSO can sign a cooperative agreement with local health facilities (both public and private) and promote service linkage with emerging health insurance schemes. 11 3.6. Even if direct food support is restricted in the SCOPSO project, the food support remained to be one of the essential services it provided the children within its limited budget. To address the huge need for food the project beneficiaries have, the project encouraged schools to develop relationships with local institutions (e.g. private or public hotels, bakeries, college/universities and individual families) interested in supporting the nutritional needs of OVC. While this is an appreciable trend, it still could not satisfy the prevailing needs of the beneficiaries. Hence, WL-E needs to work more on community mobilization to leverage resources from the community and increase the number of needy OVC to the food and nutritional support. 3.7. Educational support interventions for the OVC need to include improving the reading and writing skills of the children. Hence, WL-E needs to devise mechanisms whereby reading and writing projects can be part of the SCOPSO intervention. 12 1. INTRODUCTION 1.1 BACKGROUND In Sub-Saharan Africa, an estimated 15.7 million children aged 17 and younger have lost one or both parents, mainly due to the HIV/AIDS epidemic which accounted for 30% of the 53 million anticipated orphans from all causes. Ethiopia alone is reported to have an estimated 5.5 million orphan and vulnerable children, which is around 15% of the total child population. Of these, 16% were orphaned due to HIV/AIDS (MOH, 2007). Overall, girls suffer more than boys by not being able to attend school, having to care for others, and being forced into early marriage. All orphans and vulnerable children, directly or indirectly, are vulnerable to HIV/AIDS and other health, socio-economic, psychological and legal problems (MOWA and FHAPCO, 2010). These situations have exposed children to various life threats including poverty, hunger, child labor practices, malnutrition, illness, abuse, psychological problems and stigma and discrimination. These problems appear to aggravate as the OVC get older and are much more vulnerable to HIV infection. Cognizant of this and in response to the needs of children who are orphaned or made more vulnerable because of HIV/AIDS, the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) has earmarked around 10% of its funding for activities to improve the wellbeing of orphans and vulnerable children (OVC). PEPFAR/USAID principles and approach for OVC programming includes strengthening families as primary caregivers of children; strengthening systems to support country ownership, including community ownership; ensuring prioritized and focused interventions that address children’s most critical care needs; and working within the continuum of response to achieve an AIDS-free generation. In Ethiopia, both government and non-governmental organizations (NGOs) working from national to local levels also play a critical role in the response to children. In August 2009, USAID/E and World Learning Ethiopia (WL-E) signed a contract agreement to support the School-Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project which started in August 2009 and will end in 2014. This five-year project, aims to strengthen the ability of communities to participate actively and own the design, implementation and management of OVC support activities at schools in sustainable way. The overall objective of the project is to build the capacity of 400 primary schools to serve as focal points for OVC care and support to at least 52,000 HIV affected or infected OVC leading to increased enrolment, retention and academic performance. Despite recognition of the efforts of WL-E in its SCOPSO project, there is a need to evaluate the performance of the project - whether it achieved its intended objectives and contributed towards improving the well-being of OVC along the essential services. To this effect, PRIN International Consultancy and Research Services PLC has been granted to undertake the terminal evaluation of the SCOPSO project by USAID/E. In pursuant of its commitment, PRIN had applied a fairly rigorous methodology to generate authentic data and produce credible report. In particular, the evaluation process had paid special attention to exploring what program interventions worked and which did not work well in terms of improving the well-being of children and their caregivers in households and communities most affected by HIV/AIDS. Apart from identifying the major achievements and emerging impacts of the project, the evaluation had tried to pinpoint the major challenges faced during implementation of the project as well as the possible recommendations that may contribute to devising appropriate strategies for circumventing the challenges and ensuring the sustainability of the project's outcomes. 1 1.2 INTERVENTION MODEL The SCOPSO project has been implemented in all regions of Ethiopia since 2009. The project focused on strengthening the communities’ abilities to participate in the design and management of OVC support programs using the school as a conduit to empower and organize community members. Support to OVC through schools built upon the existing primary school organizational management system, mainly the Parent Teacher Associations (PTA). PTAs worked in close coordination with community health promoters, health facilities, local government, other parents and guardians, community volunteers, and local and international non-governmental organizations to ensure a coordinated system of service delivery for OVC. Schools were the focal point for these linkages and all development efforts targeting children and the general community. The project provides comprehensive care and support to OVC in primary schools. The project also ensures access for orphans and vulnerable children to essential services, including education, health service, psychosocial support, food support, shelter and care, and economic strengthening activities. In addition, the project builds the capacity of school community (i.e. PTAs, GACs and teachers) to identify and support OVC through improved access to education and better performance in school. The project also strengthen capacity and facilitate the function of REBs, WEOs, KETBs, school communities, WHOs and Woreda (local) administration in supporting increased access to education for OVCs. As one of its strategies to sustain OVC care and support, community mobilization activities have been carried out to raise awareness about the needs of OVC. In this regard, the School Core Group (SCG) played a major role in mobilizing the wider community and key influential actors such as religious leaders, community elders and various community based organizations. Moreover, the project provides funds to support school communities to initiate low cost approaches for income generation scheme such as, school and community gardens, tea rooms and café, mini-shops, handicrafts, etc. that primarily benefit OVC beneficiaries. In short, the following two key results were expected from the three-year (2009 – 2012) SCOPSO project: Result 1: Provision of quality, comprehensive services to at least 52,000 HIV-affected or infected OVC, with a focus on increased enrolment, retention, and academic performance; and Result 2: Strengthen School-Community Partnership in 400 primary schools to enhance their capacity to plan and manage OVC support programs. 1.3 PURPOSE OF THE PERFORMANCE EVALUATION The overriding purpose of this performance evaluation is to gain an independent view of the overall performance of the project in order to help USAID/ Basic Education Office, Ministry of Education, Regional Education Bureaus, schools, communities and other stakeholders with regard to the overall achievement of the project’s intended results. To this effect, the task evaluation of the process was devoted to assess the key results and illustrative activities, the outcomes, challenges, lessons learnt, and level of satisfaction of the key stakeholders. In particular, the final performance evaluation of the SCOPSO project was conducted with the objectives to: 2 • Determine and verify the quantitative and qualitative results (input, output and, outcome) of the project since its inception vis-à-vis the project agreement, aim, objective and results. • Assesses the project’s approach and methodology to achieve the project objective and key results; • Assesses the project’s management structure, staff composition and relationships/partnerships with schools, communities, local host government entities, NGOs, Community Based Organizations (CBOs) and other stakeholders are pertinent to achieve the projects objective and results; and • Explore the level of satisfaction on the part of the Ministry of Education, Regional Education Bureaus, Woreda Education Offices, schools, communities and beneficiary OVCs and their parents/guardians and others. In response to the evaluation's scope of work (SOW) provided by USAID/E, the core evaluation team drafted and submitted questions and strategies designed to collect qualitative and quantitative data about the performance of the project. Very useful briefing and planning meeting was conducted with USAID/E and WL-E colleagues to further refine the objectives, -the focus of the questions, and the approach –of the evaluation. 1.4 EVALUATION QUESTIONS In order to carry out the evaluation within the purposes outlined above, the following four major questions were raised: 1. How effective is the project in achieving its set, objective and anticipated results? a. What the achievements of SCOPSO project against planned objectives are as embedded in the Monitoring and Evaluation plan? b. Which services provided to OVCs are critical to improve the retention, enrolment and academic performance of OVCs? c. How effective is the project in linking the services provided to OVC to other sectors such as health and others? d. How effective is the project in addressing the needs of girl/female OVCs? e. How cost effective is the project in attaining its objective and results, particularly on cost per OVC? f. What has not been achieved and why? 2. How is the project’s approach and methodology designed to achieve the project objective and results? a. How effective is the collaboration, coordination and working relationship among the implementing partner and stakeholders such as school PTAs, local Government institutions, NGOs, CBOs and others? b. How effective is SCOPSO management structure and staff composition in terms of (i) relevance to the project intervention, (ii) Monitoring and Evaluation procedures and standards; and (iii) the overall project management environment? c. What institutional arrangement did implementing partner make to ensure sustainability of the project’s results? 3 3. What are the levels of satisfaction of the Ministry of Education, Regional Education Bureaus, Woreda Education Offices, schools, communities and beneficiary OVCs and their parents/ guardians? And why? 4. What are the lessons learned from the project? What are challenges encountered and measures taken? 2.DESIGN AND METHODS OF THE EVALUATION This evaluation was carried out on the basis of participatory approach in which the inputs of relevant stakeholders were aggressively sought and duly considered in the evaluation process. The evaluation methodology was determined by the consultant team in collaboration with USAID/WL-E. The major method of the evaluation is the qualitative method. However, some quantitative data summarized from the project annual and semi-annual reports were also used. The evaluation data were collected from both primary and secondary sources by employing different tools.The qualitative data were generated through focus group discussions, in-depth/key informant interviews, case analysis, and observation. More specifically, the data collection process involved the following: • a review of background materials, including project document, contract agreements, the 2012 PEPFAR OVC Guideline, the 2010 MoWA and FMOH Standard Service Delivery Guideline (SSDG) for OVC Care and Support Program, semi-annual and annual reports, and other training materials. • key informant interviews with government officials; • focus group discussions with OVC and guardians (program beneficiaries); • in-depth interview with implementing partners (WL-E staff and School) and PTA leaders; • guiding tools for case study development; • Performance Statistics from intervention schools and Project Offices; • School Statistics on Enrolment, Retention and School Achievement; and • Standard service deliveries checklist (critical activities). In this regard, series of meetings were made with USAID/E and WL-E staff to identify areas of emphasis in the evaluation, and agree on the design of the instruments and sample schools. Accordingly, six regions, 41SCOPSO intervention schools and 6 non-intervention schools were chosen to be samples of this study (kindly see in the Annex section). Owing to a very limited time given for the evaluation assignment, the samples were selected on the basis of a combination of purposive, convenience, and availability sampling methods. Six teams were deployed for data collection. The first team had visited North Amhara; the second team had visited South Amhara; the third team had visited Dire Dawa and Harari; the fourth team had visited Oromia; the fifth team had visited SNNP; and the sixth team had visited Afar. Each team comprised one lead researcher and two assistant researchers. Informants were selected based on their different roles in SCOPSO project implementation–coordinators from WL-E, project implementers in schools (school core groups and technical working groups), PTA members, homeroom teachers, OVC and guardians, and government officials from WEO and REB (i.e. SCOPSO focal persons). As a result, a total of 1005 informants were interviewed from 41 SCOPSO intervention and 6 non-SCOPSO intervention schools as well as from relevant government offices. 4 The data collected were summarized in Microsoft Excel. Analysis included textual classifications of the responses and highlighting of emerging themes. Four to five themes were identified as major findings in each area of the evaluation. The themes are interpreted and reported as major findings in this report. The analysis of the effectiveness of the project took the PEPFAR guideline as a basic frame of reference because the SCOPSO project clearly refers to this and other OVC guidelines in Ethiopia (kindly see SOW for the contractor, p.9). Limitations of the evaluation: Though every possible care has been taken in designing and conducting this performance evaluation; it is by no means without limitations. Due to time constraint the sample schools taken for the study is very small (only about 10%) compared to the total population. However, effort has been made to get adequate information about the project by using multiple tools to collect data from multiple sources. In addition, critical review of reports has been used to get a good picture of the overall national situation of the project. 5 3. FINDINGS This section is organized into eight sub-sections, namely, project design and relevance; efficiency; project effectiveness; systems and structure; emerging impact of the project; opportunities and challenges; and sustainability of the project. 3.1 PROJECT DESIGN AND RELEVANCE 3.1.1Project design USAID/Ethiopia, as matter of its principle, has initiated and designed the project based on the national priority needs (as identified by FHAPCO, MoH, and MoE) and the experiences it (USAID/E) gained from similar project-based interventions in Ethiopia. As noted earlier in this report, the project has six pillars called ‘the six core OVC services’. The provision of these six OVC services is believed to be crucial for boosting school performance/achievement, promotion, retention, child well-being and for reducing student absenteeism and dropout rates. As the SCOPSO project is school-based intervention, respondents indicated that schools were selected by the project focal persons in discussion with Woreda Education Office. Burden of OVC and socio￾economic conditions of community living in the nearby vicinity of the schools were few of the reasons why some schools were targeted under the project. All participants remarked that the objective of the project is associated with improving the lives of children so that children could attend their school throughout the year with better performance. In addition, in its attempt to spell out the needed areas of intervention (gaps) clearly, the WL-E had carried out a survey, which eventually served as a baseline for setting the targets for SCOPSO project. The baseline survey was carried out by its M & E Unit in 2010. The baseline survey had helped to generate information on the levels of OVC school attendance, academic performance, and dropout rates. In addition, the data has been organized to inform program implementation and management as well as to provide critical stakeholder information to support the on-going development of school-based programs designed to support orphan and vulnerable children in Ethiopia. Among the recommendations that came out of the baseline survey were: a) Improve access to education: • Mobilize community through school enrolment campaigns to identify out-of-school OVC and support them to attend their schooling; and • Strengthen tutorial and after-school assistance for enrolled OVC requiring academic support. b) Support health and nutrition: • Increase OVC access to health care and strengthening of linkages with relevant services in their respective communities; and • Provide OVC with supplementary food and nutrition; c) Support economic strengthening: • Strengthen and expand IGA activities for long term improvement (e.g. provision of training and seed capital for IGAs, including school gardens, running micro-enterprises, or linkages with micro-finance institutions). 6 d) Promote psychosocial wellbeing: • Ensure satisfaction of psychosocial needs of OVC; • Create trusting and supportive environment at schools that allows provision of individualized attention, and personal counselling of children; • Organize home visits through care providers, and conduct frequent monitoring of physical and emotional conditions of OVC; and • Train Core Care Group to provide support to other OVC care providers to help them cope with stress of their own situation. e) Challenge stigma and discrimination: • Increase HIV awareness and improving access to correct and comprehensive information about HIV/AIDS; • Train teachers to contribute in the reduction of stigma and discrimination; and • Collaborate with local authorities to protect children and their rights to property and safe place to live. f) Involve the beneficiaries: • Actively engage the OVC in the interventions that they are to benefit from; and • Facilitate access to practical social and financial skills training to OVCs to increase their chances for independent life and contribute to their household economic strengthening. Even though there can be every claim that USAID/E had designed the project based on the policy direction in Ethiopia as well as its (USAID/E) practical experience in the field; the involvement of stakeholders, especially during designing and planning of the project was limited to higher level only. For instance, it did not embrace parents/guardians/community leaders at these critical initial stages. 3.1.2 Relevance to community needs Children in Ethiopia face many challenges during childhood. As a child’s age increases, the likelihood of the child living with both parents decreases significantly. Seventy-two percent of Ethiopian children under age 18 live with both parents; 14% live with their mothers but not their fathers; 3% live with their fathers but not with their mothers; and 11% live with neither of their natural parents (Ethiopia Demographic and Health Survey- EDHS, 2011). The 2011 EDHS also reports rural children are substantially more likely to live with both parents than urban children (74% versus 58%). The same report indicated that among regions of the country, the highest proportion of children living with both parents is in Benishangul-Gumuz (75%), while the lowest proportion is in Addis Ababa (52%). Lack of parental care and support due to HIV/AIDS exposes children to increasing vulnerability, such as food insecurity and chronic malnutrition, lack of protection/shelter, lack of access to education and physical and sexual abuses. These children also face the increased burden of caring for ill parents and of stigma and discrimination. This vulnerability can, in turn, increase children’s risk for contracting HIV/AIDS. In response to the aforementioned situation, the SCOPSO project, through the PEPFAR/USAID funding, responded to the challenges by providing quality and comprehensive services to HIV-affected or infected OVC, with a focus on increasing enrolment, retention, and academic performance. Ultimately, the SCOPSO interventions enable children to form a sense of self-worth and belonging and are essential to learning, developing life skills, participating in society, and having faith in the future. 7 Substantiating the relevance of the SCOPSO project in the context of their community, discussants and key informants from all school community concurred that the project is very relevant in addressing the needs of OVC and it was well contextualized. In almost all of the schools, OVC were in very destitute situations and were not getting basic life necessities. A high number of students used to drop out of their schooling despite the efforts of the school community. The arrival of the project to the schools was considered as a miracle since it has brought meaningful and measurable changes in the lives of targeted OVC. Specifically, its pertinence in increasing enrolment, reducing dropouts, improving academic performance of target OVCs has been underscored by the respondents. Regarding this here are some representative opinions of the interviewees: “...not only the OVC but the school itself has benefited from the project. The project was very appropriate and it enlightened our knowledge on use of existing resources to support the children. Our mobilization skill is also enhanced...” SCG member of Mollae School “...our area is highly malaria-prone and our children frequently get sick. The intervention is very relevant to solve this and other deep rooted problems...” a School Principal of Wanjana Kulito Primary School in Halaba town “...had it not been due to the project, the students would not have attended their school - as simple as that...” A member of Adelo Primary School SCG Regarding relevance, the SCOPSO intervention minimizes the dropout of OVC. In light of this, a key informant from Arba Minch town education office said, “...in our town, dropout rate ranges from 20-30%. But due to the project and collective efforts, it is reduced to 10%. This is mainly attributable to the SCOPSO project....” Head of Arba Minch town Education Office Substantiating the aforementioned responses, in the third year of the SCOPSO implementation, it was reported that the dropout rate was found to be 6%, which shows a 2% increment from the first year. However, it is by far less than the national average dropout rate 16.31 (17.4% for boys and 15.1% for girls) in 2010/11, as indicated in the Ethiopia Education Statistics Annual Abstract (2011/2012). In general, at the start of the SCOPSO intervention, it was targeted to support at least 40,000 HIV/AIDS affected or infected OVC at 400 primary schools in nine regions of Ethiopia, including two city administrations. However, in its second year of implementation, the project extended its comprehensive service provision to an additional 12,000 OVCs on the same number of schools. That means 100 OVC from each school were targeted at the beginning and later the number increased to 130-150 depending on the total number of identified OVC in discussion with the donor. The Majority of the implementing partners remarked that even though the project is very appropriate and timely, the number of children targeted is low when it is seen in light of the total number of OVC identified by the school community. In some of the schools, only 50% of the registered OVC were targeted by the project. Even distribution of OVC across the schools was raised as a shortfall of the project regarding targeting; it is believed that the allocation should have been made based on proportion to OVC population size and total number of pupils in the school. In other words, the targeting did not consider the number of OVC in each school 1 Education Statistics Annual Abstract (2011-12), Ministry of Education, EMIS, Planning and Resource Mobilization, Directorate, September 2012, Addis Ababa, Ethiopia 8 and the allocation was uniform in the first year of the program. Later, in the second year, the project tried to consider OVC case load based on the target. 3.1.3 Relevance to the government strategy Ethiopia has ratified both the UN Child Rights Convention (CRC) and the African Charter on Rights and Welfare of the Child (ACRWC). The country has harmonized domestic laws and policies with the provisions of both conventions, which creates an enabling environment for improving the wellbeing of OVC2. In response to the needs of OVCs in the country, the Ethiopian government sets its own strategic plan. In its Strategic Plan II (2010/11 – 2014/15)3, one of the thematic area is strengthening care and support services to mitigate the impact of AIDS with a general objective of improving the livelihood of the needy OVC. Its specific objective is increasing care and support to needy OVC from 30 percent in 2009 to 50 percent by 2014/15. Its main strategies include strengthen the involvement of local communities in care and support; enforce the provision of standardized care and support to OVC; enhance school based OVC support; and strengthen income generation activities to sustain the program. Regarding this the key informants from Woreda Education Office remarked that the current intervention run by SCOPSO is in line with the government strategy. The government set a target that all school aged children should attend school. As a result, the support provided by the project helped the Woreda Education Offices to meet its goal in increasing enrolment of children. The currentgeneral education quality improvement package (GEQIP) also promotes school retention and promotion, which is also the specific objective set by the project. Concerning goals and gap filling, the key informants stated that quality is the primary focus of their office and it is only assured if the schools are able to support students. As the schools provide counselling and other support services to the students, the students became stable, focused and attentive in their education. This can be seen as a contribution to both access to and quality of education in the target schools. Enhancing internal efficiency (which is mainly measured in terms of dropout and repetition rates) is one of the priorities of the schools. In most of the visited schools, OVC are likely to have higher dropout rates due to lack of parental follow-up and the low economic conditions of households. However, with SCOPSO project the dropout rates of OVC decreased even in areas with cash-crops (which were used to be characterized with high dropout rates). The project closely follows the status of OVC. This also gets the attention of the government in that both OVC and non-OVC need to participate in essential services in an effort to avoid stigma and discrimination. To this end, the project increases access to education services for OVC, as there is provision of material support such as school uniforms and school supplies. Accordingly, OVC have got educational opportunities without any discrimination; and educating OVC is an important component in the social well-being of these children and in reducing poverty. According to the key informants, the school community gained a lot from the project. The contribution of life skills training, social and financial responsibility is helping not only OVC but also all students in the schools. As part of the SCOPSO project (in 2010), WL-E also has AFLATOUN’s model which contains both social and financial themes. This has helped children learn about themselves, saving, basic financial concepts, and enterprise. 2 Standard service delivery guidelines for OVC care and support programs (MOWA and FHAPCO, 2010) 3 Strategic Plan II For Intensifying Multisectoral HIV and AIDS Response in Ethiopia: 2010/11 – 2014/15 (FHAPCO and FMOH, 2010) 9 AFLATOUN puts children at the centre of their learning process and engages them with the world around them. Inso doing, AFLATOUN believes that the social and financial lessons that they receive will stay with them forever. As it was reported, this intervention involves both OVC and non-OVC students within the school. Key informants were also asked to identify the strength and weakness during implementation of the projects. They reported continued support for school attendance of OVC; continuous and strong follow-up; provision of training; active involvement on IGA; strict and transparent finance control; and continuous discussions with stakeholders are some of strengths they noted during the implementation of the project. The respondents reiterated that the SCOPSO project is very relevant and appropriate to the national priority and community needs due to the aforementioned reasons. However, they add that even though the support is well commendable, it did not address all gaps observed in the lives of the children. Some supports are very limited to a number of households. For example, an IGA support needs careful assessment, prioritization and follow-up by school community and the project. The respondents had also highlighted some of the limitations of the project executions such as disseminating educational materials like uniform after the beginning of academic year; small amount of budget [due to the limited budget of the project] assigned to IGA; and engaged few numbers of beneficiaries on food and nutrition support. 3.1.4 Participation of stakeholders Nowadays, there is no question over the value of stakeholders’ participation in the success of a project. The evaluation team could note that the project has well designed and systematically interwoven mechanisms of involving its key stakeholders at various levels. The participants had also confirmed that the USAID/E, as a matter of its principles, is engaged in projects of national priority which involved due consultation of relevant stakeholders. Initially, the mission had consulted FHAPCO, MoE, MoH, and others before inviting proposals from implementing partners. Following the consent of the key stakeholders, the implementing partner (i.e. WL-E) was selected and advised to work closely with the key stakeholders at federal, regional, and local levels. Accordingly, the WL-E had been implementing the project in consultation and full involvement of the key stakeholders all the way from Ministry of Education to Kebele Education and Training Board as well as community/household levels. For example, PTAs are involved in follow-up activities of WL-E in the school; working in close contact with school SCOPSO coordinators; committee member at school level; selecting OVC beneficiaries; and ensuring that the material provision to OVC are fair and effective. In addition, the SCGs are involved in managing the funds and the programs efficiently and effectively. The regional education bureaus and the MoE are involved in annual progress reviews, joint planning exercises, and observation performances on the ground. 3.2 EFFICIENCY 3.2.1 Input The program was initially planned for three years: August 15, 2009 through August 14, 2012. As it is indicated in the contract agreement modification# 3 the total fund for the three years activities amounts to US$9,999,978.This SCOPSO project was funded by USAID-Ethiopia, and World Learning-Ethiopia is the recipient of the fund and utilized the funding for its school based OVC project. Through this funding 10 WL-E provides direct support along the six essential services which include education support, psychosocial support, food and nutrition, health care, shelter and care, and economic strengthening. Moreover, WL-E utilized the funding for indirect supports such as capacity building including training to ensure and provide quality of services. Besides, WL-E also allocates budget for sub-grant. It was mentioned that targets for OVC were clear and, therefore, it was easy to measure impact/success. Figure 1: OVC Program Costs by Line Items (for the three years) Inputs in the form of labour-time contribution from the school (e.g. time of SCG, support from local officers in M & E, etc.) are all essential contributions to the success of the project. 3.2.2 Efficiency of the Project There are a number of critical issues for decision makers, both within the World Learning/Ethiopia SCOPSO project and donor agency-USAID/E. National estimates put the orphan population in Ethiopia at 5,459,139 as of 2009, of whom 855,720 are orphans due to HIV and AIDS (MOH, 2007). Thus, the SCOPSO project supported 52,000 orphan and vulnerable children which accounted for 6% of the orphaned children due to HIV/AIDS at the national level. It is therefore important that decision makers focus attention on the provision of services that will augment the quality of life of targeted children. Given the anticipated OVC situation, the efficiency with which services are provided is also equally important. In this evaluation, rather than using a cost-per-child to judge a program’s efficiency, it might be more helpful to look at what percentage of funds reaches project supported schools, either through direct material support, capacity building, or as a sub-grant. In this regard, a key question for decision makers relates to how OVC resources in SCOPSO project are being allocated across services. This evaluation indicates that the largest percentage of resources is being allocated to education (29% of all the direct financial support provided to schools), followed by income generating activities (17%), food and nutrition (10.5%), and then health care (9.6%). Relatively small percentage of resources has been spent on psychosocial support (2.3%). On the other hand, the evaluation identified that some percentage of the resources is being allotted to stationery, recreational and sanitary materials (17.9%), which would fall under such three essential services as education, PSS and shelter and care respectively according to the operational definition for each essential services. The evaluation also identifies school incentive award (amounting to 13.6%) which would necessarily fall under the economic strengthening. While this study cannot recommend an ideal allocation of resources, it is nonetheless critical that decision makers consider this allocation and ask whether this is likely to achieve the greatest possible impact for OVC. For example, should donors be spending more funds on services such as education? Are there alternatives to relatively expensive services such as economic 13% 6% 5% 0% 3% 4% 50% 19% Personnel Fringe Benefits Travel, Transportation, and Per Diem Supplies Equipment Sub-Grant Other Direct Costs Indirect Costs 11 strengthening? Decision makers may wish to focus more attention on the cost per child reached with a specific service rather than an overall cost per child reached. They may also be interested in knowing what they need to budget in order to reach a specified number of OVC with the current package of services. In this evaluation, the direct financial support made per child reached is ETB479.93 (US$25.8) per annum. Decision makers and program managers need to consider, however, if another combination of services or efficiency gains could contribute to either a less expensive cost per child reached or improvements in the quality of life of OVC. For example, are the economic strengthening and food and nutritional interventions essential for the livelihood and quality of life of the child? Should additional SIA/IGA or food items be added to the economic strengthening or nutritional interventions? The analysis of cost drivers also raises questions for decision makers and the way in which they allocate their resources. In this case, the study found significant variations in the cost per specific essential services, depending on the scale of the program. For example, food and nutrition, and economic strengthening interventions were serving more than 20% and 22% of the targeted OVCs, respectively, while others were reaching the overall targets. The evaluation here suggests that for education support, it is much more efficient to fund programs which reach large numbers of children. Programs providing food and nutrition [target 10%] and economic strengthening [target 25%]services appear to have lower costs with the more children reached, as indicated in the annual reports; however, there may be diseconomies of scale as larger numbers of children are reached. On the other hand, there are indirect supports activities that are implemented by SCOPSO to promote quality in education, psychosocial support, income generating activities for OVC and their guardians. In addition, basic and refresher trainings are provided for school core group members, teachers, school principals, Area coordinators, project officers, and other key strategic actors to improve capacity, efficiency and quality. Some of the major trainings areas include psychosocial support and life skills trainings; business skill training for trainers; and VSLA methodology and transfer assets. In terms of efficiency gains, it is important for decision makers to use this evaluation report to further assess where potential increment or reductions in project budget could be achieved per essential services. Therefore, this information can help program managers and decision makers to make informed decisions in developing and scaling-up effective strategies, ultimately reaching larger numbers of OVC with greater efficiency. 3.2.3 Costs by Service Area This evaluation report provides an analysis of costs disaggregated by service area. That means only the total costs (direct support made to schools) by service area are presented here. However, cost estimates under each service area could not comprise of the costs associated with the labor, travel and transport that go towards provision of the respective services. On the other hand, it was not possible to associate a resource with only one activity. In short, estimates were made regarding the contribution of each resource to respective service areas only based on direct support made to schools. The unit costs were derived on costs per OVC reached. Figure 2 and Table 1 present aggregate and unit costs per service category respectively. However, in the unit cost per service area IGA and food support apparently inflated, as the number of beneficiaries for these two service areas is very small. Figure 2: Aggregate costs per service area as a percentage of direct costs made to schools 12 Table 1: Cost per child per service category Service Cost per child (in USD4) Material Supplies 13.08 School uniforms 21.27 Medical support 7.01 Food Support 37.44 Source: Annual SCOPSO Progress Report (Oct. 2011 – Sep. 2012), World Learning Ethiopia, 2012 In general, in this evaluation report measurement of the costs of services per child is not a sufficient estimate of cost-effectiveness, since it doesn’t provide information on the quality of services provided to beneficiaries. On the other hand, the cost estimates focused only on those direct financial supports provided to all schools rather than considering all the economic costs; as economic costs include the actual value of all resources used, including the imputed value of voluntary labor, donated goods, and the like. However, the evaluation report serves as an input for decision makers and program managers to consider further study to estimate cost effectiveness and cost benefit analysis; and it also serves them to think about and see the intervention areas where they are going to focus in their future scale up program. 3.3 PROJECT EFFECTIVENESS As stated earlier SCOPSO has been designed to attain the two key results, including: Result 1: Provision of quality, comprehensive services to at least 52,000 HIV-affected or infected OVC, with a focus on increased enrolment, retention, and academic performance; and Result 2: Strengthen School-Community Partnership in 400 primary schools to enhance their capacity to plan and manage OVC support programs. Accordingly, the effectiveness of the project has been examined in terms of the attainment of these results and this section has been devoted to explain the performance of the project against the six core service components of the project (viz., Education, Psychosocial Support, Food and Nutrition, Healthcare, Shelter and Care, and Economic Strengthening) and capacity building efforts. Hence, the 4 Currency exchange rate: 1 USD = 18.60 ETB 18% 30% 10% 11% 14% 17% 0% Stationery, recreation& sanitary materials School uniforms Medical support Food support SIA IGA Furniture for PSS Center 13 practices, achievements, and innovation attempts recorded and credited to the SCOPSO project in each of these component areas are presented in this part of the evaluation report. 3.3.1 Education Support PEPFAR programming guidance encouraged OVC programs to support efforts to reduce educational disparities and barriers to access among school-age children through sustainable “systemic” interventions and by: (1) ensuring children have a safe school environment and complete their primary education; (2) promoting access to early childhood development programs; (3) ensuring personnel create child-friendly and HIV/AIDS- and gender-sensitive classrooms; (4) strengthening community￾school relationships, including partnering with out-of-school programming; (5) consider supporting post￾primary school programming and especially the transition for girls from primary to secondary school; and (6) implementing market-driven vocational training only when previous lessons learned are integrated into intervention designs. In light of this, SCOPSO had promoted school registration and attendance for OVC, with specific activities including organization of enrolment campaign, provision of education supplies, provision of tutorial support, and provision of school uniforms or alternative clothing to OVC. SCOPSO also provided school supplies including exercise books, rulers, pens, pencils, and erasers. In addition, SCOPSO also provide stationery materials to schools for office use. SCOPSO facilitates vocational training especially for older OVCs and those in child headed households. This training includes hand crafting, knitting, and running small businesses. The training is linked to income generating scheme. In the second year of the project, it was reported that all targeted OVCs received assistance from the SCOPSO project to attend school. That is, it was reported that all 52,000 (24,806 Male, 27,194 Female) targeted OVC beneficiaries received assistance. However, in its first year of the project, a total of 38,087 OVCs which accounted for 95% among the 40,000 targeted OVCs had received assistance. In the first year of the project, it was reported that the average promotion rate was 87percent for grades 4 -7; and 4percent dropout rate for all intervention schools. However, in the third year of the SCOPSO implementation, it was reported that the promotion rate was found to be 86 percent (85% for boys and 88% for girls) and the dropout rate was found to be 6 percent. A closer look at the data reveals that the promotion rate recorded in the third year resembled to that of the preceding year but the dropout rate had shown a 2 percent increment. Further, although the aggregate dropout rates for both sexes happened to be more or less the same, the dropout rate for boy appeared to be higher in lower grades while that of girls appeared to be higher in the upper primary grades. Some of the major factors that contributed to the improvement of students' academic achievement were found to be continuous tutorial support, commitment of teachers, provision of psycho-social support, distribution of educational materials, and other targeted supports rendered by the project. In this regard, the key informants of the study had also noted that supports given to OVC need to strengthen the development of basic computation and reading skills at primary level. Among the factors that contributed to the school dropout, the dominant happened to be the demand for child labour at home. In this regard, the evaluation team had learned that the SCG, TWG, and PTAs have been working hard to convince the guardians to reduce the burden of household chores for the students. 14 Qualitative interview result shows that guardians and OVC reported that one of the most satisfactory supports was education support. Receiving of school uniform and exercise books (8-10), pen/pencils/eraser (2-4) annually was well appreciated by the beneficiaries. According to discussants, children who used to remain at home have started their schooling due to the support they get from the project. They also remarked that one of the inhibitory factors for low enrolment in the past was inability to meet educational expenses. But due to the project, not only the burden is lifted but also the school in discussion with KETB have decided to exempt OVC families from annual community contribution, which is 50-100 birr per year usually paid at the beginning of the academic year. All emphasized the significance of change brought about by the project in this regard. Similarly, a mother expressing her relief she felt due to exemption from the school contribution. She said, ‘....even the exemption from annual contribution by itself was a relief for most of OVC families. I, myself was intending to stop sending my child to school since I have no income and a single parent custodian of four children...’ a mother in a community Picture 1. OVC beneficiaries of SCOPSO project in Awash Arba, Afar. One older OVC also noted he is motivated to attend his schooling; and he said, “… we are now motivated to come to school every day because of SCOPSO” an older OVC 3.3.2 Psychosocial Support PEPFAR programming guidance advocated for OVC programs to prioritize psychosocial interventions that build on existing resources and place and maintain children in stable and affectionate environments through: (1) parents and family support programs; (2) peer and social group interventions; (3) mentorship programs; and (4) community caregiver support. 15 Within the SCOPSO project, the service includes provision of recreational services, individual or group counselling services, and life skills education. Accordingly, in the first year of the project, a total of 1,002 teachers were trained in psychosocial support and life skills education. In its second year, the project had provided refresher training on areas of counselling and life skills education for a total of 1,894 teachers. In the second year of the project, it was reported that All 52,000 OVCs received psychosocial support that includes counselling and recreational support. Out of this, a total of 20,141 (9,648 male, 10,498 female) OVCs received individual counselling services. Life skills education was provided for a total of 15,271 children, which is an increase by 14.8% over the first year achievement. In the third year, all of the target beneficiaries (52,000 OVC) had received at least one of the key psychosocial services provided by SCOPSO. The services were provided by 1002 trained school teachers at 400 PSS centers. In particular, among the beneficiaries, 52,000 OVC had received counselling service while 5,199 OVC had attended life skills education program. Moreover, the project had established counselling rooms in the 400 schools to provide psychosocial support services for the needy OVCs. SCOPSO also furnished these rooms with tables, chairs and shelves to make them suitable for counselling services. These centers provided space for open communication among children and teachers, and serve not only OVC but also other children who need psychosocial support. SCOPSO supported working groups aimed to identify the major problems experienced by OVC beneficiaries and to provide encouragement. SCOPSO also encourages OVCs to participate in recreational and sport activities, and provides life skills education to the needy OVCs. Qualitative interview result show that majority of key informants expressed that psychosocial support aimed at enhancing OVC school performance and keeping their emotional wellbeing. The project in collaboration with schools has established separate counselling rooms for PSS service. The school community constructed or avail free space and the project furnished those rooms. Trained teachers are delivering the counselling with close assistance from PSS sub-committee (of three people). The project also provides life skills trainings (Sport for life and Beckons) and avail sport materials such as balls and nets. The staffs had appreciated the commitment of school teachers in delivering the support and they attributed the documented effectiveness to the efforts of school community. But linkage with community based counselling such as elders and faith leaders were not documented. The schools has been delivering PSS to OVC through school based individual/private and group counselling. According to the discussants, the intervention is effective as it addressed the emotional well-being of children with insignificant budget. Concerning HIV AIDS prevention, guardians reported that they are getting information from the local health workers (HEWs) and in some cases from mass-media and communications. They also have information regarding the education provided at school level since their children told them what they learnt in schools. Some of the guardians participated in community conversation programs in the past. Similarly, OVC reported receiving of health education on HIV, reproductive health, sanitation and others. Most of the OVC discussants successfully mentioned three ways of HIV transmission and prevention. 16 3.3.3 Food and Nutrition The evaluation team had learned that SCOPSO supported working groups had initially identified a total of 10% of all project OVC beneficiaries who were most in need of food support to continue their schooling. In the first year of the project, SCOPSO disbursed food/nutritional support funds amounting ETB 3,777 per school to all 400 schools. This had served a total of 3,680 (1,884 boys, 1,796 girls) OVC, which accounted for 92% of the target beneficiaries for food support. In the second year, the project supported an additional 1,969 (959 boys, 1,010 girls) OVCs. However, the number of orphan and vulnerable children in need of food and nutritional support exceeded the resources available for intervention, as the beneficiaries increased by 8.6%. In the third year, 10,647 OVC (5,417 girls) had received food and nutrition support services from the project's emergency food support fund, school-based micro-enterprises, and/or through referral linkages with individuals and private/public agencies. The findings in this respect revealed that the achievement was more than double the initial target (set for the year) in the project, which was to serve only 5,200 OVC. The Chief of Party and the Monitoring and Evaluation (M & E) expert from WL-E have pointed out that the major factor contributed to the overachievement of the service was credited to a high level commitment and extra effort demonstrated by the SCG members in various schools. Further, SCOPSO supported schools and communities played key role in supporting OVC food needs by collecting in-kind food contributions such as cereals from farming communities; and establishing relationships with private restaurants to provide food for OVCs; including using IGA proceeds to cover food costs for needy OVCs, and by schools’ internal contributions made by individual teachers and students. Qualitative interview results show that key informants explained about the process for food and nutritional support in their community. As they revealed, one of the care and support service components is food support to identified OVC. In this regard, schools have been trying to support selected number of students through direct grain support. Limited fund is allotted from the project and it is directly transferred to school bank account, and the SCGs do the planning and prioritization of students that need food support. In some schools, it was reported that OVC themselves were involved in the process of selection of students for food support since the money earmarked for food support was very minimum and only covers limited number of OVC. Here, according to SCG members, the project targets to reach at least 10% of children for food support but so far due to the magnitude of the problem, additional number of children were receiving the support and yet the dilution effect is there. It is worth to note that in some schools it is reported that the support was delivered on or before the hunger seasons. One of the key informants reported that SCOPSO food and nutritional service is very essential for the needy OVC. He said, ‘ ...we are feeding some children that are not fed by their guardians. Few of them came to school with empty stomach and cry here. Even some of them skipped both dinner and breakfast...’ a teacher from Goro school in Arba Minch Guardians and OVC also reported that selected households have received 20-25 kilos of maize grain per year. Selection of beneficiaries was made by the SCG in collaboration with PTAs; and most discussants remarked their dissatisfaction on the limited number of beneficiaries, as the project targeted to support 10% of the overall OVC targets due to its limited budget. On the other hand, one of the food support focal person reported that their project extends its food support beyond the SCOPSO targets. He said, 17 ‘...we have supported 20 kilos of maize grain for nine students who are out of SCOPSO target using the fund generated from school land...’ Jore school food support focal persons OVC also reported their involvement in the selection process. In few schools, all targeted OVC were involved in the selection process for they know who needs the support. Eleven years old OVC reported that they are involved in the selection of beneficiary for food support. He said, ‘...we were involved in the selection of OVC for food support. We know who is eating at home or who is not. So we say, ‘he is not eating so pick him for food’ ....’ an 11 years grade 4 OVC Of all sampled schools, few schools follow a different strategy. The SCG took the initiative and prepare food for selected students throughout the year. An average of 30 students had been receiving the support for the past three years. The food support area was one of the supports that did not satisfy the needs of all children. Shortage of budget and high cost of food led to dissatisfaction among some beneficiaries. The SCG use the same filtering criteria like age, sex, household status. Those OVC who were very young, female and were from the poorest of the poor got the support. Since the budgeted amount from the project was limited, the school community, especially teachers contributed some money and feed some OVC that were not targeted by the project. It is worth to mention here that PEPFAR programming guidance restricted direct food support and encouraged programs to leverage food security assistance from other partners. PEPFAR guidance provided for weaning foods and other nutritional support for children less than two years of age, nutritional counselling, information dissemination, and systems supports such as policy and technical guidelines development or coordination to promote food access. The guidance excludes school feeding programs and broad based food-assistance for all school children. However, owing to its design, the SCOPSO had managed to support considerable number of children and it encouraged schools to develop relationships with local institutions (e.g. private or public hotels, bakeries, college/universities and individual families) interested in supporting the nutritional needs of OVC. 3.3.4 Healthcare SCOPSO healthcare component includes preventive and curative healthcare services including age appropriate reproductive health, HIV prevention education, personal hygiene and environmental sanitation education; and improving access to health care services including obtaining waivers for free medical treatment for needy beneficiaries; and covering the costs of medical treatment in the event of emergency illness. Activities consisted of indirect supports, such as referrals and linkages to child healthcare. To address this healthcare service, SCOPSO supported working groups have established linkages and relationships with health care service providers including health posts and Kebele Administrations. Using its linkages with Government partners, the project relied upon existing health programs to provide child healthcare. Teachers, health extension workers and invited health officers working in the nearby health centers provided general health education to increase the health awareness of OVCs. 18 In the first year of the project, SCOPSO provided healthcare services for a total of 34,008 (16,516 boys, 17,492 girls) OVCs, which accounted for 85% of the target beneficiaries. These OVCs also attended health education including reproductive health, HIV prevention, and hygiene and sanitation. A total of 177 (64 boys, 113 girls) OVCs were also referred to health facilities for medical care; and 20,000 girls were provided sanitary materials, including soaps and underwear. During the year each school had also received an emergency medical support fund totalling ETB 3,700, which was allocated for the treatment of emergency illness of OVCs. In the second year of the project, with the SCOPSO project a total of 1,474 (698 boys, 776 female) OVCs obtained medical treatment; and a total of 1,336 (604 boys, 752 girls) OVCs received medical attention through referral. The school based IGA proceed also supported medical services for 82 (26 boys, 56 girls) OVC beneficiaries. To this end, WL-E had disbursed ETB 6,240 for emergency medical funds for each school. In the third year of the project, all 52,000 beneficiaries had received medical service ranging from minor check-ups to sending them to referral hospitals. Of these, 105 OVC (75 boys and 30 girls) were referred for higher level health services for illnesses that could not be treated at school health centers. In relation to this, the evaluation team had interviewed onebeneficiary, whose story is narrated as under, I am BeyehuAlemayehu who is 13 years old and attending my 5th grade class at WajanaKulitona Primary School in Halaba town. My father died when I was a kid in Amhara Region. When my father died we had nothing for our daily bread. As a result, my Mom had travelled to Halaba to look for her relatives. After sometime, she was able to meet her relatives and she came back to Amhara region to take me to Halaba. Then after,my mother got married to AtoAlemayehu and gave birth to two daughters. Now we are three children at home.AtoAlemayehu has no permanent source of income. So my mother is forced to be engaged in a daily labor in Blate nursery site to feed us and cover all required costs for my education. To supplement our family's income for bread, I was also serving people for little income. In doing so, I faced an accident by bicycle and injured on my upper left eyebrow and my neighbours took me to the nearby clinic and got treated. However, it was not as such simple to cure it and the infection got expanded to the inner part of my left eye. Finally, my left eye became completely out of sight. As a result, I faced serious difficulty to attend my class. But being in such hardship situation, I ranked 7th in Grade 4. Then my neighbours again took me to the clinic. After the check-up, the health worker told them that it was not simple to treat and even it may expand to my right eye if immediate medical attention is not given. So he advised them to take me immediately to a better hospital. But for a family like ours, who are getting their daily consumption from daily laborer mother, how can one imagine her take me to a better hospital? Finally, the USAID funded SCOPSO project in our school and the community had supported me to go to an eye specialist and got treatment. Now after replacing my left eye with an artificial one, I'm very well and attending my education at the same school. Now, I hope I will be ranking 4th to 6th in Grade 5. So I would like to use this opportunity to thank the USAID/SCOPSO, the community, WanjaKuliton Primary School Core Group members for supporting me during such hard time. Bayeh Alemyahu Qualitative interview results show that SCOPSO arranged provision of health care services to registered OVC. All OVC are entitled medical care whenever the need arise. Guardians are advised to use their own money and all expenses reimbursed upon presentation of legal expense receipts. School 19 level implementing partners assures that the medical care is a well-organized care and gives satisfaction to the children. With regard to the difference between OVC and non-OVC in accessing health services, key informants revealed that OVC have medical budget from WL while the Non-OVC get medical treatment via family expense. But few mentioned their reservation regarding the approach that direct refunding of expenses for it has created some problem since some households might not have money to take their children to health facility. Few also reflected on the restricted use of left over money to other components. In addition to medical care, OVC and other students are receiving health education. Trained teachers and health extension workers provided health education on personal hygiene and environmental sanitation, RH and HIV and AIDS prevention. As key informants noted, the objective was to keep children at school and reducing absenteeism and dropouts due to ill health. As stated by the project staffs, health education on different subjects was given to all students with the help of local health providers. The education also incorporates TT vaccination and VCT promotion. The earmarked emergency medical fund is also covering medical expenses of children through reimbursement procedure. Sanitary materials were also provided to students. Even though the project has tried to link OVC with the existing public health facilities for free medication, so far little efforts exerted in this regard. The staffs mentioned the intention of working on the area in the final year of the project. 3.3.5 Shelter and Care In this domain, PEPFAR guided programs to support access to adoption and fostering services, temporary shelter for children in transition, and family reunification when possible. Guidance also included home maintenance support to child or youth-headed households and encouraged innovative community responses to provide care in personalized settings when family options are not available. Given the number of OVCs, particularly in the country, and their complex needs, the most effective responses place families, households and communities at the center of interventions. In this component of the project, SCOPSO with the support of Kebele Administrations, KETB and the communities provide safe shelter for needy OVCs. SCOPSO supported SCGs facilitated OVCs access to shelter, clothing and care. SCGs held discussions with the local community members on the need for foster services for neglected children. That is, SCGs ensures vulnerable children to get at least one adult foster who could provide them with the necessary support they need. However, this SCOPSO project did not focus resources on shelter or home maintenance, due to limited funding and limited ability to leverage support. In this component, in the first year of the project a total of 3,669 (1,475 boys, 2,194 girls) OVC beneficiaries, which is 9.2% of the total targeted OVCs, were provided with shelter and care support. About 237 (100 boys, 137 girls) OVCs were integrated with their relatives; temporary shelter was secured for 298 (127 boys, 171 girls) OVCs; and two boys have been placed in foster care. In addition, clothing was provided for 869 (449 boys, 420 girls). In the second year of the project, SCOPSO supported working groups in collaboration with Kebele administrations and the community reintegrated 48 (22 boys, 26 girls) OVCs; found temporary shelter for 141 (71 boys, 70 girls) OVCs; placed 140 (72 boys, 68 girls) OVCs with foster families and provided 20 clothing. A total of 29,379 (2,185 boys, 27,194 girls) OVCs received SCOPSO assistance for personal hygiene supplies. In the third year of the project, 27,994 OVC (26,854 girls) had received at least one of the services included within the shelter and care component of SCOPSO project (i.e. family integration, provision of temporary shelter, foster care service, house repair, clothing and underwear provision are the main services considered in this intervention). This accomplishment was reported to be more than twofold the target set for the year, which was 13,000 OVC. However, since distribution of underwear has been included within this indicator, the service inflated the number of OVC served under the shelter and care component of the project. Qualitative interview results indicate that apart from clothe/school uniform support by the schools; the engagement of the project on mobilization of house renovation was very limited and only a few number of OVC received housing support due to limited capacity at all levels. Very few number of OVC houses were renovated with the involvement of students and local community members. Covering house rental expenses for OVC was almost non-existent as it is reported by very few OVC though discussants remarked the need. But the guardians and OVC appreciated the cloth support since it is the major burden of families in availing the resource at the beginning of academic year, as the SCOPSO work voraciously on clothing and care. Sample key informants also depicted that all students studying in schools that have set distinctive set of clothes have been getting school uniform support in the past three years. Those without such regulations, OVC received casual clothes. Schools hand over cash to guardians and guardian’s present evidence for the proper use of supported money. Few reported that the current support does not consider the inflation. In addition to this, the project delivered sanitary and cleansing materials such as soap, oil and sanitary pads and underwear for girl students. On the other hand, one of the key informants noted that they have constructed a house for OVCs through mobilizing the local community. He said, ‘...in Goro school, the SCG let OVC to live in a house constructed for income generation since the children were living in the streets...’ 3.3.6 Economic Strengthening Under this SCOPSO component, both OVC and guardians are supported to have stable income. SCOPSO implemented its economic strengthening activities at the school, community, and individual or household levels. Major approaches within this core project component includes transfer and building of tangible assets, organizing and establishing Village Saving and Loan Association (VSLA), implementation of AFLATOUN child social and financial education; and enhancing school internal income through school￾based income generating schemes using the school incentive awards (SIA). This component of the SCOPSO project directly addresses the critical issue of sustainability beyond the life of the project. In the first year of the project, the first round of school incentive awards were distributed to initiate school-based income generating activities, including managing tea rooms within school compounds, small shops, gardening and animal fattening. Schools had also mobilized resources from several other sources using information collected through community service mapping activities. In addition to the 21 aforementioned activities, 470 OVC guardians and older OVCs had engaged in economic strengthening activities through linkages and referrals created by SCOPSO. Picture 2. OVC beneficiaries of SCOPSO project through economic support from Durame, SNNPR. In the second year of the project, it was reported that 400 schools proposals were reviewed and school incentives awards with a total value of ETB 2,952,000 was distributed to all schools. SIAs were used as seed money to run school based IGAs that include but are not limited to mini shops, cafeterias, gardening and animal fattening. Most of SCOPSO supported schools have transferred their cafeteria to be rented or generate income to support the needy OVCs. Several municipalities were also cooperate in providing land and permits to construct shops and gardening spaces for income generating activities. Besides, SCOPSO disbursed ETB 12,048,920 for economic strengthening activities that include the implementation of VSLA and transfer of assets in all regions. The asset transfer5 intervention is used for child-headed OVC households; OVC with elderly or seriously sick parents or care givers; OVC who have no caregivers; as well as in areas where the project team has determined that VSLA cannot properly work. From the aforementioned disbursed money, schools transferred tangible assets worth ETB 1,643,217 and benefiting a total of 2,265 (711 male, 1,554 female) OVCs and their guardians. In its second year SCOPSO, all selected 103 schools in eight regions implemented VSLA activities and a total of 2,190 (285 male and 1,905 female) guardians were organized to form 118 VSLAs. However, the project accomplished only 55.69% through VSLA and asset transfer, as the project targets 8,000 OVC and their guardians based on its need assessment results. Besides, in the second year of the project 318 school saving clubs have been established and the number of club members reached 27,155 children (14,142 girls). Out of these, 41.9% are SCOPSO supported OVC, 15% non-SCOPSO supported OVC and the remaining 43.1% are non-OVC students. Thus, a total of ETB 65,589(~US $3,858) was saved by all AFLATOUN members. In the third year of the project, 25,562 OVC (15,260 girls) and 6,863 parents/guardians (5,138 women) were reached with economic strengthening services, which was found to be by far more than what was 5 Assets transferred in rural areas include cattle, goats and sheep; while in urban areas guardians acquired materials for small retail business activities, including table tennis and bicycle rental service. 22 originally planned (13,000 beneficiaries only). The service components that do fall under this intervention include establishment of school-based micro-enterprises, engaging guardians and older OVC in VSLA, and undertaking asset transfer for the destitute OVC as well as social and financial education service rendered for OVC. Regarding the implementation of the child and financial education, SCOPSO increased to 376 schools. A total of 906 AFLATOUN clubs having 38,951 (20,396 girls) members reached. Of these clubs, 853 had regular savings amounting to ETB 294,075.54 (~US $15,810.51) as of June 30, 2012. Around 126 clubs established social enterprises addressing various social issues of interests to the children. Mixed reaction was reflected by both group discussants concerning the economical strengthening programs run at household level. Those beneficiaries who received the direct support/asset transfer and member of village saving and loan associations reported their happiness and effectiveness of the support. The support includes providing seed money for guardians; purchasing sheep/goat for students; renting shops/containers; photocopy services and student savings. They mentioned that the activities brought attitude change and OVC families have started working and saving instead of waiting for others to reach in cases of emergencies and financial difficulties. A number of guardians and OVC remarked that they have the belief that the future is brighter and they are motivated. All remarked they have received trainings and orientation on business development and saving at the beginning of the support. But only very few reported their involvement with the local microfinance institutions. Students and guardians mentioned that PTAs and SCG follow the proper use of support (though in some schools, role of PTAs was limited). However, those discussants that are not addressed by the IGA support questioned the selection criteria and requested the schools to consider their appeal. Interview results also show that the project carried economic strengthening at the schools and household level with the intention of supporting households and schools for sustained capacity and continuation of care program. Key informants also realized that the support was carried out to ensure sustainability of care after the end of the program. Key informants also explained about the economic strengthening support in four major components: school enterprises, asset transfer, and village saving and loan association as well as AFLATOUN. School enterprise: These activities focused on creating income sources for the schools under the program. So far the area focused on construction of shops/houses for rental, strengthening school cafeteria for OVC care, fattening, school gardening and plantations. Most of the school, though it varies, have started generating their own income through these and other means. Schools are also contributing their own matching fund from school/block grant. Asset transfer: The asset transfer strategy was utilized based on careful analysis of the context and/or based on the preferences of individual OVC households. Accordingly, goats, coughs, sheep, etc. were provided to Households and follow-up of proper utilization was done by SCGs and PTAs. A guardian and two OVC beneficiaries reported how the economic strengthening support is changing their lives. Here are their opinions: “....I have been supported one sheep three years ago and now they are three. I am enjoying since I have assets...” a 46 years old guardian “....my mother received a goat two years ago and now, we have eleven goats as the goat delivered triples three times....” a 14 years old male OVC “...my mother got 1000 birr loan a while ago. She has returned the loan back after doing some 23 business. We have used the profit for household consumption...” a13 years old female OVC Village Saving and Loan Association (VSLA): the village saving and loan association were organized by the schools and OVC HHs save some amount for a limited period of time before getting any help from the project. The staffs reported that those guardians that showed proper management and commitment received seed money from the project. The project also provided business training and counselling along with market assessment to selected beneficiaries. So far graduated VSLA members were few and in recent past the project took a step and gave seed money to VSLA members. However, linkage with microfinance institution was limited. Regarding direct support instead of revolving fund, some staffs provide such justifications as legal accountability issues, school mandates, government policy, and current global best practices. Government policy regarding micro financing activities and the current local and global best practices indicates that revolving fund schemes are not preferred methodology to do such business due to problem of their viability in relation to complications in management of the funds such as very low collection of loan repayment, misuse, etc. AFLATOUN: Child social and financial education is a training model that promotes social contribution and effective financial management. Children learn about themselves, children’s rights, how to save, basic financial concepts, and enterprise. They organized themselves into clubs and do some social work and all including OVC have started saving some amount of money. A number of children have opened bank account and others are saving using the school saving box. Picture 3.AFLATOUN day celebration in South Wollo. 3.3.7 The situation of non-intervention schools as indicators of the effectiveness of the project Survey conducted at the six non-intervention schools revealed observable differences from intervention schools in terms of services to the OVC. In almost all the schools observed it was learned that: 24 • Most of the school communities knew what SCOPSO is and what the project is doing in other schools. For instance, one school principal has requested clarification from the Woreda Education Office regarding criteria used to select schools for SCOPSO project. The school principal reported that the office’s response was not convincing. • Even though attempts have been made by the schools to identify a number of OVC in the past, it was reported that the OVC are facing several problems which could have been somehow solved through the kind of assistance which SCOPSO provides. Among the problems reported to have challenged such children were lack of enough food, education materials and clothing as well as the need to assist their families (both in the field or domestic work). • Most OVC who managed to get to school could not make it to successfully complete their schooling – dropout and repetition were reported to be very high among such children. For instance, in one school alone out of the identified 15 OVC, 11 dropped out during the first semester. The school in collaboration with KTB had tried to get back the students, but failed to manage family resistance. It was learned that in that school the remaining four OVC continued their studies through the support of teachers. • Cross learning among SCOPSO and non-SCOPSO schools is limited as observed during the discussion at the schools. The focus is given to direct support areas and little is known about services such as PSS, counselling and local resource mobilization. From these points it is possible to note that SCOPSO-intervened schools are relatively effective in terms of serving the target OVC. The experience of SCOPSO could have been utilized by the Woreda Education Office and its local partners (e.g. PTA, Kebele Education and Training Board, etc.) to help the OVC children by looking beyond direct assistance. Community mobilization, internal resource generation, academic support by teachers and many more opportunities could have been utilized. Lack or weaknesses over these again flash back on to SCOPSO in terms of its institutional impact (on Woreda Education Bureaus). 3.3.8 Capacity Building The second major result area that was planned to be pursued under SCOPSO project includes 'Strengthen School-Community Partnership in 400 primary schools to enhance their capacity to plan and manage OVC support programs.' In this regard, the facts observed from project implementation areas (target regions and schools) have revealed encouraging performance of SCOPSO. Within the context of OVC services, “capacity building” means strengthening the social service system, including the technical and operational capacities of its individual actors and organizations6. PEPFAR programs should prioritize within their country context the following capacity-building and systems￾strengthening interventions: investing in efforts to build strong leadership and governance; pursuing strategies to strengthen the social service workforce; supporting strategies to improve financing for social service systems; strengthening information management and accountability mechanisms of the social service system; and supporting coordination and networking within the social service system. It was also noted that all key stakeholders involved in providing service and support to OVC should ensure that users of the Standard Service Delivery Guidelines, at all levels including federal, regional and local are trained in the application of the Guideline. The stakeholders should also ensure implementers 6 Guidance for Orphans and Vulnerable Children Programming (PEPFAR, 2012) 25 have technical, financial and managerial capacities necessary to successfully facilitate and provide quality services to the needy OVCs, as per the Guidelines. Substantiating the aforementioned facts about capacity building, in its first year implementation of the SCOPSO project, it was indicated that project implementation within school communities began with providing a series of start-up orientation workshops for key strategic actors including PTA representatives, KETBs, school principals, cluster supervisors, WEOs, Zone Education officials, RSEBs officials, representative from community-based organizations, religious leaders, Woreda HAPCO, administrators, and Women’s Affairs officials. In total 2,559 (2,025 male and 534 female) participated in the workshop. Those orientation workshops resulted in school personnel becoming familiar with project goals and objectives, and strategies. The aforementioned Key strategic actors in the SCOPSO project also gained such valuable skills as proposal writing, financial management, and how to effectively support and care for OVCs. Moreover, SCOPSO has developed different OVC supported training manuals to contribute to the process of making the SCOPSO project operational, by building skills of key strategic actors – SCG and PTA members, KETB representatives, community members, teachers, supervisors, and school principals at the Woreda, school and community levels, to enable them to facilitate and provide quality services for orphans and other vulnerable children. The reported trainings are summarized here under per the project implementation year. In its capacity building efforts, SCOPSO reported that in the first year of the project, training on psychosocial support and child status index was provided to a total of 948 (570 male and 378 female) teachers and cluster supervisors with the aim of preparing teachers to provide psychosocial support service to the needy students. Besides, life skill training was given to a total of 751 (416 male and 341 female) teachers to encourage them to provide similar training to their students in their respective schools. In the second year implementation of the project, as part of technical support training on implementation of individual OVC care plans, recruitment criteria and implementation and accounting of school IGA and AFLATOUN activities were provided for a total of 1,894 (1,306 male and 588 female) school core group members to manage the OVC supported activities. Moreover, training on VSLA methodology and training guideline was provided or a total of 41 (37 male and 4 female) project staff including Area coordinators, program officers, field assistants, M & E staff, finance staff and senior managers of SCOPS project. These project staff in their part cascaded the training to a total of 2,190 (285 male and 1,905 female) guardians on business skills, regular saving, and rules and regulations of the associations. In addition, business skill training of trainer (ToT) was provided for a total of 37 (33 male and 4 female) SCOPSO staff with the objective of cascading for VSCL where members to have a better understanding and exposed to the business environment. In the third year of the project, refresher training for a total of 1,435 (952 male and 483 female) school core group members with a focus on the psychosocial needs of OVC, financial management, economic strengthening, community mobilization, and fundraising. Moreover, a two day stakeholder review meeting was organized in all project sites and a total of 398 (330 male and 68 female) participants attended. Participants were drawn from regional, Zonal and Woreda education offices. In general, SCOPSO could provide the key stakeholders with the major capacity building support (technical competence as well as managerial skills) to run the project according to the PEPFAR expectation. 26 3.3.9 Clients’ Impression about the Project Through this performance evaluation it had been learned that the project has been highly appreciated by almost all stakeholders. For example, the key informants interviewed were asked to identify whether the services provided to them or their children are critical to improve school enrolment, retention and school performance of OVCs. In this respect, custodians and OVC strongly emphasized that all supports were effective and critical for students to attend their school. The medical support is helping them to avoid absenteeism due to illness; the food support helps them to give focus and attention in their studies and not to drop out for the sake of getting food. The counselling services, sanitary materials, education, and IGA all contributed to the wellbeing of the beneficiary children. Of all the supports, however, the education support has a leading role in determining the schooling of a child. Guardians remarked that the support by itself initiated many families to send their children to school. However, the number of OVC who dropped out of school shows slight increment (~2%) at the third year of the project. Akin to the guardians, OVC stated that the educational material support and food support are the two key interventions along with counselling and medical care to achieve improved retention, performance and enrolment. Similarly, a male guardian from Adela area expressed his due regard to the support made by SCOPSO as, “...what can I say, all supports are very helpful for their schooling. Today they are in school because they have the books and the uniforms, because they are healthy, and because they are eating something....” a 42 years old male guardian from Adelo area The key informants were also asked to forward their overall impression about SCOPSO. All discussants remarked that they are well impressed by the project. They liked most the education and health support as it has blanket coverage for all targeted children. Limited number of beneficiaries for IGA and food support along with untimely distribution of support such as educational material after the start of academic year or holidays were few of the points that were singled out as gray areas by the project. Similarly, a mother supported by SCOPSO expressed her impression as, “...Our overall impression is very grateful to the project. It gives us a chance to live again and to start seeing things from different perspectives. Today not only we but our children also trying to save....” a mother of two An OVC student also expressed his impression by saying, “had the assistance from SCOPSO was not available, we would have been at home, without education.” an OVC student The key informants were asked to depict the specific reasons for project effectiveness and the challenges encountered. Here are summary of the representative responses: 27 Comprehensiveness of the project: the majority of the informants reported that the holistic approach followed by the project is the key reason for successful and effective implementation of the project. Activities such as education support, clothing/uniform, medical expenses coverage are few of the most effective activities. Volunteerism and commitment of school community: the other reason pointed out by discussants is the devotion of school teachers to the care program. It is surfaced out that the SCG members work day and night to arrange service for children. Those teachers who are supposed to go home after their shift stayed behind and use their personal time for OVC care. Notable commitment of WL project staffs: in all areas, it is observed that good relationship between project staffs and school community is the one of the foundations of the effectiveness. Discussants appreciated the dedication of WL staffs. Even though the aforementioned points had contributed to the success of the project, some reported the following as inhibitory factors: • Budget limitation: Even if direct food support is restricted in the SCOPSO project, the food support still remained to be one of the essential services the project provided the children within its limited budget. To address the huge need for food on the part of the project beneficiaries, the project encouraged schools to develop relationships with local institutions (e.g. private or public hotels, bakeries, college/universities and individual families) interested in supporting the nutritional needs of OVC. Nevertheless, the resources available from all these sources could by no means satisfy the available gap. • The other limitation was work burden on school community. Some schools are working with limited number of teachers and teachers are overstretched. Regarding the burden of teachers, a school principal from Kulfo said, ...some teachers are providing 26 credits per week and it is difficult to assign them for extracurricular activities. This resulted in shortage of volunteer teachers...” a school principal from Kulfo • Weak community attitude toward child schooling in some cases: even though improvement in communities’ attitudes towards schooling has been generally reported; there are instances where community members are not fully supportive. As a result, in (around) some schools enrolment is yet to be appreciated and dropout is still a challenge. Target schools in collaboration with the KETBs have been trying to convince parents to send their students to school and to support their continued study. 3.4 SYSTEM AND STRUCTURE Among the issues treated under this section are beneficiaries’ selection mechanism, project coordination and implementation structure, community mobilization, and monitoring and evaluation. 28 3.4.1 Beneficiary Selection Mechanism Before the commencement of the project, WL-E developed selection criteria for targeting the OVC in accordance with PEPFAR/USAID guidelines, and shared it to the targeted schools to identify and select the needy orphan and vulnerable children. It is also reported that being single or double orphan, vulnerable due to various reasons such as being HIV positive, from poorest family and disability were few of the key OVC registration criteria applied during selection process of the OVCs. In assertion of this, an interviewee from Oromia Education Bureau reiterated that the OVC selection was made based on four major criteria that were applied as per the following order of importance: (1) loss of both parents due to HIV/AIDS; (2) loss of father due to HIV/AIDS; (3) loss of mother due to HIV/AIDS; and (4) children from destitute families who cannot afford to pay for schooling of their children. The selection of the OVC is usually carried out with a committee drawn from Woreda Education Office, PTAs, and School Core Groups. In addition, the project staff from WL-E explained that the project uses the national agreed definition for selection of OVC. The SSDG is commonly used as a reference to determine OVC and the criteria were communicated to all schools prior to targeting. The staff also stressed that only a child is considered from a household and priority was given to female, young and the poorest of the poor. Some regions had tried to do wealth ranking. In this regard, the Office of the U.S. Global AIDS Coordinator (OGAC) and PEPFAR issued OVC Programming Guidance in July 2006 that stated “Vulnerability” of OVC because of any or all of the following factors that result from HIV/AIDS: is HIV-positive; lives without adequate adult support (e.g., in a household with chronically ill parents, a household that has experienced a recent death from chronic illness, a household headed by a grandparent, and/or a household headed by a child); lives outside of family care (e.g., in residential care or on the streets); marginalized, stigmatized, or discriminated against; and is at risk of contracting HIV. Despite the efforts made by WL-E and the selection committee to choose the right beneficiary, the evaluation team had witnessed varieties of complaints from the guardians and project implementing partners. For instance, the Director of Denebe Primary School had stated the difficulty of applying the selection criteria as follows: ‘...We have learnt selection criteria from the project and we have carried out the registration accordingly. However, we frequently countered a challenge to pick targeted fixed number of children out of the registered OVC since the prioritization criteria was not clear and direct...’ by Denbe primary school principal 3.4.2 Project Coordination and Implementation Structure During the first year of implementation, the project made significant strides toward achieving these results, and it also played a major role in supporting the government of Ethiopia’s concerted efforts to mitigate the impact of the HIV pandemic on the nation’s most vulnerable children. Thus at the commencement of the project, WL-E built a strong foundation for successful implementation by formalizing key relationships with partners – RSEBs, RSHBs, WEOs, WHOs, schools, communities, Woreda HAPCO, Woreda and Kebele administration and individuals; and by establishing management systems in each regions. 29 Regarding coordination and management of the project, in the first year annual progress report it was indicated that there are 6 coordination offices and 17 project offices established. Besides, WL-E had established School Core Groups (SCGs) at each of the 400 schools; and these SCGs in turn formed smaller sub-groups specializing in the six core technical areas of the project interventions (education, psychosocial support, health, food and nutrition, shelter and care, and economic strengthening). SCGs are comprised of teachers, school principals, PTA representatives, and OVC beneficiaries. Moreover, WL-E also builds their capacity by its project staff on key topics including coordinated guidelines for OVC, community mobilization, the provision of psychosocial support to OVC, and life skill education. Its intention is to improve their technical capacity to deal with the multiplicity of factors facing AIDS￾affected children and their families and communities; and to provide quality service to project beneficiaries. However, few key informants revealed that some PTA members didn’t get training in this regard. As PTA members were replaced every two years, many of them were trained and left their assignment. Thus, those newly assigned PTA members took the responsibility to serve in the project but didn’t get any training. Substantiating the aforementioned report, key informants revealed that at branch offices there are area coordinators, program officers and data encoders. At school level there are School Core Group (SCG) comprising KETB, PTA, school principals, cluster supervisors, religious leaders, students; and six technical working groups who are volunteers. As the key informants described, in most of the schools, the school core group, which is chaired by the school principal and has its own secretary who is also PSS focal person, is the highest decision making organ in the implementation structure. It is reported that all communications from WL project focal persons pass through the SCG and the SCG assigns specific subcommittees to plan and handle the issues. PTAs, as a member of the SCG, work closely with the other members in identifying OVC for support and follow up of support process. In addition to the PTAs, district/town education office and KETB work closely with the schools. Regarding coordination and adequacy of workers, majority of discussants and key informants emphasized that all their work is based on pure volunteerism. It is remarked that the work of delivering care to OVC has brought additional burden to teachers. In some schools, due to the high number of students and shift schedules, teachers are already stretched. Discussants put forward that the project staffing is limited and almost all of the service delivery was done by school teachers with close follow-up of project staffs. A project officer is expected to cover an average of 22 schools which is a burden. One of the key informants described the work burden on few members of SCG. As the key informant said, ‘...the scarce resource in the care and support program is volunteer human resource. We can talk about other resources such as finance, but in the past we have learned that volunteers are crucial to the success of care program. We are working evenings and weekends to cope up with the burden...’ a SCG secretary from Kulfo schools With regard to the role of the people involved in the project coordination and management, key informants revealed that he role of area coordinator is coordination of overall SCOPSO project activities; communication and/or cooperation with stakeholders; facilitation of program delivery; monitoring and oversight of the project, and conduct review workshops and admin head of the area. While the role of project officers include supervise and provide technical support for target schools; capacity building for school teacher like training, coaching and mentoring; preparing monthly, quarterly 30 and annual reports; planning annual activities of the project; follow up performance and work closely with schools communities. Besides, as key informants mentioned some of the specific achievements of the project includes reduced dropout, increased academic performance of OVC, and increased the capacity of schools and guardians. While they also mentioned some of the challenges which includes shortage of vehicles; shortage of time; and some school principals being reluctant to discharge their responsibilities. To summarize the implementation of the SCOPSO project structure, one of the project staff indicated that the project structure includes Chief of Party (COP), Deputy Chief of Party (DCOP), program manager (technical specialists/IGA/AFLATOUN), M & E, and finance at the WL head quarter level. At Area Coordination level there are area coordinators and project officers; and at the school and community level, there are school principal, SCG, OVC and their households. 3.4.3 Community Mobilization Empowering communities to mobilize and utilize existing resources will help generate ownership and sustained action to support OVC. Programs should ensure that communities have the necessary support to take responsibility for addressing the needs of OVC. Such an approach will work towards ensuring ownership of the services by the community and hopefully enhancing the sustainability of services and support7. In response to the aforementioned facts, as indicated in the first annual progress report of the SCOPSO project, community mobilization is one of its strategies to sustain OVC care and support. Community mobilization activities have been carried out to raise awareness about the needs of OVCs. As a result, improved community participation and involvement in providing care and support for OVCs is emerging. The SCG played a great role in mobilizing the wider community and influential actors such as religious leaders, community elders, and various community-based organizations (CBOs). However, interviews with key informants showed that so far the project has exerted little effort to link the existing school based interventions with community based efforts in securing food support to children. The only linkage set was use of the school cafeteria for limited school feeding programs. Mobilization of community resources to meet shelter and care needs of the children was also limited for there are few documented efforts in this regard. As far as the essential services are concerned, key informants were requested to identify specific services that are effective as compared to one another. Concerning effectiveness of IGA and economic strengthening, the key informants reported that the IGA was well planned and initial discussion and training on saving, business management and proposal/market analysis was carried out. In this regard, though the support is very limited to a number of households and guardians, it is one of the effective interventions brought about by the project. The IGA geared towards schools also an effective strategy in securing some sort of income for OVC care after the end of the project. So far the IGA includes construction of shops and school cafeteria and use of the rental money for OVC services, animal fattening, school gardening and plantations. As per the key informants, the project only covers medical expenses of OVC upon presentation of legal receipts. Some students have card used to access free medical treatment and this was done by the 7 Standard Service Delivery Guidelines for OVC’s care and support programs (MOWA and FHAPCO, 2010) 31 working group in collaboration with Kebele officials. This is a successful example from the project where OVCs referred to health facilities for their medical needs. One of the effective interventions, as described by implementing partners, is educational support. The support held a number of students to return back to schooling and perform well in their studies. The most common reason for not sending children to schools was household capacity to fulfil scholastic materials, uniforms and payment of annual community contributions. The decision made by the schools to exempt OVC households from annual contribution along with the provision of educational materials including uniforms to students and close follow up of OVC status by school community have led to an effective intervention. As a result, enrolment, retention and promotion have significantly increased as reported by school principals and SCG members. “...in our school, dropout rate is very high. However, the dropping out of beneficiary OVC is very less compared to the non-beneficiaries. For instance, last year out of the 50 students who dropped out of school only 4 were OVC. Due to the intensive follow-up and support, we managed to keep our OVC in schools and those that discontinued their study were due to relocation of family members.” School principal of Kulfo School in Arabminch To increase access to food, the key informants depicted, referral and linkage of OVC to hotels, cafeterias and volunteer individuals is practiced in few schools in rarely cases. Such arrangement itself is a challenge as some OVC were requested to engage in difficult and time taking jobs. Some key informants also revealed that there are school feeding programs in their community. One of the best practices of the SCOPSO project is leveraging community resources and braid funding streams to support programs and activities aligned with their common vision. As a result of community awareness raising and mobilization efforts, a total of ETB 8,451,336.39 (USD$467,958.82) was collected in cash, kind and labor from the community in the period of October 2009 to June 2013. Figure 3: Community contribution in cash, kind and labor Source: World Learning Ethiopia Community Resource Mobilization Report (October 2009-June 2013) 3.4.4 Monitoring and Evaluation Effectively monitoring and evaluating (M & E) the implementation of OVC programs is fundamental to understanding and tracking the changing nature and scope of vulnerability facing children. This is in addition to measuring the quality and impact of interventions on children’s wellbeing. They are critical 30% 25% 45% Cash Material Labour 32 elements of successful programmes, because they: determine programme effectiveness; identify and address problems; show impact; gather evidence of activities and results; strengthen fiduciary responsibility and accountability; and show transparency. It was also noted as the most important contribution to the organizational capacity and program success. The monitoring and evaluation of programs, however, demands much more detailed accounts than simply counting the number of children served. As indicated on the PEPFAR Guidance for OVC Programming (July 2012), quality monitoring (routine tracking of inputs, activities, and outputs) and evaluation (using data to assess effectiveness, relevance, and impact of achieving program goals) provide the evidence and essential information for strategic planning, program improvement, accountability of funds and effort, and advocacy. A monitoring and evaluation systems should document both quantitative and qualitative data and facilitate the collection of lessons learned and best practices. In response to the aforementioned facts, the present evaluation team has observed that World Learning/Ethiopia established an M & E system and had developed a comprehensive plan for M & E and reporting on progress, achievements, results and impact of the SCOPSO project. Appropriate indicators were also identified in order to verify the achievement of the set targets for SCOPSO project. World Learning/Ethiopia had also conducted a baseline study by the end of February 2009 with the goal of gathering baseline data to facilitate the evaluation of the effectiveness of comprehensive care provided under SCOPSO intervention on OVC school attendance, academic performance, and dropout rate. In addition, its purpose is to inform program implementation and management, as well as to provide key stakeholders information to support the on-going development of school-based programs designed to support OVC in Ethiopia. As part of its monitoring and evaluation activities, the M & E unit had assessed child status 3 times (one per each year) using a customized Child Status Index (CSI) tool to determine changes observed in children’s wellbeing. As indicated in the 3rd CSI assessment, positive changes have been observed in all the five core interventions areas. The improvement overtime shows that in comparison to the previous two assessments, in the 3rd assessment the percentage of beneficiaries movedfrom ‘bad’ and ‘very bad’ to ‘good’ and ‘fair’ situations as per the definition of the CSI tool. The results are summarized in the Table 2 below and it shows substantially increment in the ‘good and fair’ situation. Table 2: Incremental percentage of scores of ‘Good’ and ‘Fair’ in CSI assessment (over time) Core Intervention Areas June 2010 March 2011 March 2012 Food and Nutrition 46.8 64.6 76.4 Shelter and Care 50.0 64.5 76.4 Health 44.6 66.8 79.6 Psychosocial 63.7 78.2 86.1 Education and Work 59.2 73.5 82.4 Source: Annual SCOPSO Progress Report (Oct. 2011 – Sep. 2012), World Learning Ethiopia, 2012 3.5 EMERGING IMPACTS The major outcome of the project (i.e. change during the life time of the project), as discussed earlier in the report, is improvement in the educational situation of the targeted OVCs particularly by way of increased enrolment, reduced dropout and improved academic achievements of these children. The fact 33 that the children started to practice the health related lessons they get through the project can also be among the important outcomes of the project. Given the very nature of education that it is “self￾propelling,” it is possible to assume that such outcomes would have far reaching impacts on the OVC, the schools and the communities in terms of how the OVC handled. Therefore, the emerging impacts of the project can be outlined as follows: • The OVC can have significant influence in their social environment by modelling success in their educational achievement. • Communities’ attitudes towards OVC improved. The usual belief that the OVC are hopeless and have no means to better living has shown significant improvement. This situation, if properly nurtured, can lead to multifaceted empowerment of the children. • At some schools, (school) communities started feeding children who are not embraced by the project out of their own means, • School-based income generation started in many cases to support needy children (particularly the OVC), • Inculcated a sense of voluntarism among the local teachers and community members to serve the OVC. 3.6 OPPORTUNITIES AND CHALLENGES There are opportunities that can be considered to sustain the outcomes of the project. There are also challenges the project confronted. Here are a list of such opportunities and challenges: A) Opportunities • Supportive policy landscape in Ethiopia to care for disadvantaged children including those who are orphaned or vulnerable, • The issuance of Education Sector Response to HIV/AIDS Policy and Strategy by the Ministry of Education. • Ethiopia has already adapted standards for OVC services which can be used as a reference in the service provision of the project. • Presence of traditional community leaders (elders and religious fathers) whose support can be sought to influence the community members, • Supportive partner institutions such as health, education, TVET, etc. offices that can be used as local resources in respective areas or domains of OVC services. B) Challenges • High number of OVC against limited budget. This in fact is a direct outcome of the rampant state of poverty in all places of project intervention. • Volunteer teachers are over stretched due to work overload. • Though the project arranged the after-the-fact (reimbursement) procedure for medical services, the fact that the guardians lack financial means even to pay the reimbursable money limited access to the services. There is no means to use credit system. 34 • Turnover of trained teachers and principals due to transference to other places, appointment to public posts or change of job. • Even though most SCG members are committed, reluctance had been observed on the part of a few core group members. • Problem/shortage of land for IGA. In addition to this challenges, several drawbacks internal to the project have been identified: there is no evidence that indicates the design of the project was participatory; lack of clarity in prioritization of the OVC to be taken as beneficiaries of the service scheme; connection of the project activities to the target service centres (e.g. connection of the OVC health service to local health service centres) is not as strong as expected; sometimes delay in service delivery (e.g. stationary distributed after the school year starts) and logistic limitations (one vehicle for 56 schools). 3.7 SUSTAINABILITY OF THE PROJECT OUTCOMES Planning for sustainability should begin early and continue as a priority throughout the life of a project. The PEPFAR goal of building the long-term sustainability of OVC interventions also requires planning for the transition and for the continuity of service delivery to and by the community. Through the evaluation process, it has been ascertained that the project has taken care of this. As a sustainability strategy, almost all of the school communities mentioned that they are preparing themselves to use the established income generating schemes as a source to meet at least some of the basic needs of the children after the end of the project period. Rented shops, school cafeterias, land secured from school communities for banana plantation and school gardening such as pepper seed breedingwere few of the IGAs that the schools are planning to rely on. In addition, some also reported their intention to mobilize resources from the local faith-based and community-based organizations. Some are intending to contact known merchants and business persons and raise some amount for OVC care. The key informants also recommended that income generation activities need to be strengthened to ensure productivity of the beneficiaries thereby ensuring self-reliance which in turn significantly contributes to the sustainability of the care and support services. One of the female teachers who is member of the School Core Group remarked the following regarding their commitment for the sustainability of the project: ...Now we understood how we can generate income, how to mobilize the community, etc... Therefore, we are confident that we will continue the support after the end of the project. a female teacher member of SCG The key informants also mentioned that using the established school clubs, there is a plan to mobilize resources from community members. So far their engagement was limited to holidays. After the end of the program, there is a plan to scale up the mobilization to cover expenses related to educational materials support. Institutionalizing the support through SCG was also mentioned to be another activity considered to ensure sustainability of the project. In general, sustainability of the project has been taken care of through several in-built mechanisms: • Connection of the service delivery to schools, a public institution principally concerned with the education and development of the children. 35 • The fact that the project provided holistic support (i.e. comprising the six components) also helps ensure sustainability of the project. • Use of support from local community resources (e.g. health extension workers to train teachers about health care services), though not adequately used. • Change of the local community’s view/perception of the orphaned children – tendency to see that such children can learn and grow if supported. As a result some communities started to provide support from their own means, besides the support the project provided. • Local capacity development: schools, community representatives and all other part-takers were capacitated to plan a project, and manage implement of the plan as well as to monitor and evaluate progresses. • The fact that the project takes economic empowerment (e.g. provide economic means and train the OVC) can also be considered as one important element or mechanism to make sure sustainability. A drawback to be mentioned at this point is that the project has not started implementing its exit strategy. The project coordinators indicated that that is planned for 2013. However, until the time of data collection (mid of 2013) there is no visible effort regarding the implementation of the exit strategy. This deserves serious attention. 4. CONCLUSIONS SCOPSO implements its interventions at three levels: child, household and community. At the child level, SCOPSO primarily provides direct support on essential services – education, psychosocial, health, food and nutrition, and shelter and care, and economic strengthening supports, including life skills education to OVCs through its school intervention. Of all the supports, the education support has a leading role in determining the schooling of a child. The educational material support and food support are the two key interventions along with medical care to achieve improved retention, performance and enrolment. However, the food support did not satisfy the needs of all children, as only less that 20% of OVC served for food support. Besides, the engagement of the project on shelter and care was very limited, as only very few OVCs were supported; however, the project delivered sanitary and cleansing materials such as soap, oil and sanitary pads and underwear for girl students. At the household level, SCOPSO developed appropriate income-generating activities for OVCs and their guardians. The economic strengthening support was carried out to ensure sustainability of care on the part of beneficiaries. Its mechanisms include provision of assets as per the preference of OVC households as well as through careful analysis of the contexts to improve their lives. However, only less than 22% OVC households were served for IGA support. At community level, SCOPSO strengthened community capacity to support OVCs and their guardians mainly through building the capacity of key stakeholders including Woreda HAPCO leaders, zone and Woreda education officials, healthcare providers, KETB and PTA representatives, CBOs and religious leaders, and Women’s Affairs Officials by facilitating training to identify the needs of vulnerable children and deliver appropriate services to OVC and their guardians. The project also provided training, coordination and material support in order to improve the coordination and management of the program. School incentive awards were also provided to initiate school-based income generating activities to finance programs for orphaned and vulnerable children. 36 Comprehensiveness of the project; volunteerism and commitment of school community; sense of ownership on the part of the community; and notable commitment of WL project staffs are among the main reasons for project effectiveness. As reported above, SCOPSO project has done fairly well in all areas identified for this terminal evaluation. The changes aspired to be brought about at OVC, institutional and community levels have already started to be seen. The educational situation of the targeted OVC improved, schools learned to manage service delivery for OVC and communities learned how to handle OVC. Therefore, even if it is difficult to claim that the project could achieve all its targets, it is possible to note that the work performed is remarkable. 5. LESSONS LEARNED We, the performance evaluation team, do hope that readers can learn from reading the whole report. However, we would like to highlight the following for similar project-based interventions in the future: • The geographic converge of the project is immense. However, it was possible to run it as a project due to the highly structured organization in place. Particularly the use of SCG and focal coordinators for each service is an essential decision the project made. Therefore, similar big projects can learn from this. • Volunteerism is the other lesson the project taught. Where the life situation is generally pressing for teachers, it is difficult to thinks that projects of this kind would get someone who can voluntarily commit his/her time and energy for the project. This particular project has taught the possibility. • A related lesson that can be learned from the project is that teachers and all other people working among rural communities can be considered and used as development agents. When particularly the issue is awareness building and attitude change, these resources have much to offer. • There is a general assumption that ‘the poor lives in the present.’ As a result many think that poor people do not want to educate their children because they want to use their children to win daily subsistence. The present project demonstrated that even the poor communities can educate their children when they are provided with a little support. • The holistic approach the project followed within its capacity is also another aspect to draw lesson from. It practically demonstrated that change at grass root level is not only about education or health or supply of specific services. It is often the result of the integration of intervention in all the major areas of services. • There is huge demand against the limited resource the project could provide. This situation over stretched the resources without adequately addressing the basic needs of the target children. USAID/World Learning could have limited its coverage area (e.g. pilot on 200 schools instead of 400). There is much to be learned from this situation for future projects. 37 6. RECOMMENDATIONS Based on the findingsand results of SCOPSO to date, including satisfaction at individual OVC, school and community levels, the evaluation team recommended that the program need to be strengthened and scaled up. However, some points have to be considered in the future intervention for management mechanisms. The evaluation team has suggested the possible measures to be taken against the key actor as follows: 1. The Government (MoE, RSEB, WEO) 1.1 As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by encouraging and developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, both the federal and regional governments have to encourage those NGOs that are willing to work on OVC-related interventions to learn from such a good practice and devise a strategy to leverage the local resources by involving relevant stakeholders. 1.2 As it has been reported in the findings section, the success of the SCOPSO is partly attributed to the concerted efforts of school communities (viz., teachers, SCGs, PTAs, KETB, etc.) and other local partners' staff who provided voluntary service. In light of this, the evaluators recommend that the Government of Ethiopia (GoE) has to promote the idea of voluntarism and create a culture of voluntary service to support OVC in educational settings/schools. As part of such a step, there should be negotiation with the WEO as well as the schools to reduce workload of teachers who commit themselves for such voluntary support. Such voluntary support could extend to charity clubs so that older members of the club support the younger OVC after proper training. This has added advantage of teaching compassion and charitable gift. 1.3 In order to ensure the continuity of the student learning and achievement, SCOPSO-type intervention needs to be introduced at secondary school level, particularly up to Grade 10. This helps those children who do not have any kind of support to complete their general secondary schooling and enhance their likelihood of gainful employment in the future. 1.4 One of the SCOPSO's major activities that fall under educational support intervention is provision of tutorial support for the target school children. In this regard, the SCGs had strongly recommend that SCOPSO’s intervention should also be closely aligned with Early Grade Reading and Writing Project so as to address the learning difficulties/needs of OVC as they are less likely to get the necessary home-and school-based remedial support. To facilitate this, RSEBs in consultation with other relevant partners, have to devise appropriate and feasible strategy in the shortest possible time. 1.5 In order to ensure sustainability of the project's outcomes, MoE and RSEBs need to consider earmarking school grants for the schools proportionate to the number of OVC they enrol. 38 II. USAID 2.1 As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, USAID needs to encourage its implementing partners to plan for interventions which include strategies to leverage local resources by involving relevant stakeholders. 2.2 The SCOPSO project is a multi-dimensional intervention that requires the involvement of actors drawn from various sectors, such as health, agriculture, trade and industry, cooperatives, and so forth. However, the connection of the project to line sectors was found to be not as strong as expected. As a result, the children failed to get supportive services they had to from such public sectors (e.g. health). This could have been resolved had the project closely worked with those line offices at management level. Therefore, USAID need to encourage its implementing partners to form strong management relations in any similar project interventions in a way that there will be strong inter-sectoral integration. 2.3 In order to ensure the continuity of the student learning and achievement, SCOPSO-type intervention needs to be introduced at secondary level, particularly up to Grade 10. This helps those children who do not have any kind of support to complete their general secondary schooling and enhance their likelihood of gainful employment in the future. Therefore, USAID needs to allocate resources to promote SCOPSO-type intervention at secondary schools. 2.4 The SCGs had strongly recommend that SCOPSO’s intervention should be closely aligned with Early Grade Reading and Writing Project so as to address the learning difficulties/needs of OVC as the OVC are less likely to get the necessary home-and school-based remedial support. USAID can promote this by connecting SCOPSO-type interventions to its READ Ethiopia projects. III. World Learning 3.1 As it has been observed from the facts of this study, communities had shown considerable interest in supporting OVC-related interventions in different means (cash, labor, & material). This effort needs to be further strengthened by developing a sense of ownership among community actors such as community leaders, religious leaders, CBOs, PTAs, local school communities, and so forth. In this regard, WL should learn from such a practice and devise a strategy to leverage the local resources by involving relevant stakeholders. 3.2 The SCOPSO project is a multi-dimensional intervention that requires the involvement of actors drawn from various sectors. However, the connection of the project to line sectors was found to be not as strong as expected. As a result, the children failed to get supportive services they had to from such public sectors (e.g. health). This could have been resolved had the project closely worked with those line offices at management level. Therefore, WL needs to form strong management relations in any similar project interventions in a way that there will be strong inter-sectoral integration. 3.3 There is a need to check the phase out strategy not to make that pull out disruptive. The remaining time of the project should be devoted to safe transition of the support from WL-E to Communities so that the communities as well as the schools can make sure that the children continue their education after the phase-out of the project. 3.4 Now that the OVC are in school. The project is also approaching its phase-out stage. It is therefore commendable that WL-E considers conducting follow up or tracer studies after phase 39 out of the project. The output of such tracer studies should be able to inform on the proportion of children who could continue to the secondary level of education and those who continued to work their business that was initiated by SCOPSO. 3.5 The project approach for reimbursing medical expenses somehow dissatisfies the beneficiaries because they do not have enough to pay in advance. Therefore, SCOPSO has to devise appropriate and feasible strategy that facilitates easy access to the needed medical service for the beneficiaries. For example, SCOPSO can sign a cooperative agreement with local health facilities (both public and private) and promote service linkage with emerging health insurance schemes. 3.6 Even if direct food support is restricted in the SCOPSO project, the food support remained to be one of the essential services it provided the children within its limited budget. To address the huge need for food the project beneficiaries have, the project encouraged schools to develop relationships with local institutions (e.g. private or public hotels, bakeries, college/universities and individual families) interested in supporting the nutritional needs of OVC. While this is an appreciable trend, it still could not satisfy the prevailing needs of the beneficiaries. Hence, WL-E needs to work more on community mobilization to leverage resources from the community and increase the number of needy OVC to the food and nutritional support. 3.7 Educational support interventions for the OVC need to include improving the reading and writing skills of the children. Hence, WL-E needs to devise mechanisms whereby reading and writing projects can be part of the SCOPSO intervention. 40 REFERENCE DeeDee Yates, Christopher O’Connell, Nicky Davies, and MuyiwaOladosu (2008).President’s Emergency Plan for Aids Relief Track 1.0: Orphans and Other Vulnerable Children (OVC) Program Evaluation, Global Health Technical Assistance Project EDHS (2012).Ethiopia Demographic and Health Survey 2011, Central Statistical Agency, Addis Ababa, Ethiopia Ministry of Education (2012).Education Statistics Annual Abstract (2011-12), EMIS, Planning and Resource Mobilization, Directorate, September 2012, Addis Ababa, Ethiopia Mishra, Vinod, and SimonaBignami-Van Assche (2008).Orphans and Vulnerable Children in High HIV￾Prevalence Countries in Sub-Saharan Africa: DHS Analytical Studies 15. MWA and FMOH (2010).Standard Service Delivery Guidelines: For Orphans and Vulnerable Children’s Care and Support Programs, Ministry of Women’s Affair and Federal Democratic Republic of Ethiopia Ministry of Health, Addis Ababa PEPFAR (2012), Guidance for Orphans and Vulnerable Children Programming, The U.S. President‘s Emergency Plan for AIDS Relief Save the Children (2010). National Plan of Action for Orphans and Vulnerable Children: Multi-sectoral workshop on legal and policy frameworks protecting children, Pretoria, South Africa. Scott Nancy (2010). End-of-Project Evaluation Report Faith-based Regional Initiative for Orphans and Vulnerable Children (FABRIC), Boston University OVC-CARE Project, FHI UNAIDS (2000).Costing guidelines for HIV prevention strategies, Joint United Nations Program on HIV/AIDS, Geneva UNICEF (2006).Africa’s Orphaned and Vulnerable Generations: Children Affected by Aids, The United Nations Children's Fund. USAID (2009).Contract ID Code AID 663-C-00-09-00422 Technical Assistance for School-Community Partnership serving HIV/AIDS affected Orphans and Vulnerable Children (OVCs) at Primary Schools in Ethiopia World Learning Ethiopia (2010).Annual Progress Reports of SCOPSO, October 1, 2009 – September 30, 2010; October 1, 2010 – September 30, 2011; and October 1, 2011 – September 30, 2012. 41 ANNEXES ANNEX I: SCOPE OF WORK USAID/Ethiopia Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project (Contract Agreement Number 663-C-00-09-00422-00) SCOPE OF WORK November, 2012 I. EVALUATION TITLE Final Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) II. PERFORMANCE PERIOD Evaluation preparations should begin in early January, 2013 and to be ended by January, 2013. During this time the selected consultants will develop instruments for the evaluation and conduct in-country field work on/about beginning of January 2013 on/about end of January, 2013. Final revised draft report should be available by end of January, 2013. III. FUNDING SOURCE USAID/Ethiopia IV. PURPOSE OF THE EVALUATION The overriding purpose of this performance evaluation is to gain an independent view of the overall performance of the project in order to help USAID/ Basic Education Office, Ministry of Education, Regional Education Bureaus, schools, communities and other stakeholders with regard to the overall achievement of the project’s intended results. Based on the implementation results of SCOPSO, USAID/Ethiopia is also interested in learning more about what worked, what did not work and challenges in terms of the education of orphan and vulnerable children at schools. The evaluation will also help to learn what has been attained and the challenges faced during implementation for future similar project design and implementation in line with the Mission Country Development Cooperation Strategy (CDCS), Development Objective 3: Improved Learning Outcomes, IR 3: Increase achievement in basic education particularly in early grade reading. Specifically, the evaluation will: 42 1. Determine and verify the quantitative and qualitative results (input, output and, outcome) of the project since its inception vis-à-vis the project agreement, aim, objective and results. 2. Assesses the project’s approach and methodology to achieve the project objective and key results ; 3. Assesses the project’s management structure, staff composition and relationships/partnerships with schools, communities , local host government entities, NGOs, Community Based Organizations (CBOs) and other stakeholders are pertinent to achieve the projects objective and results; 4. Explore the level of satisfaction on the part of the Ministry of Education, Regional Education Bureaus, Woreda Education Offices, schools, communities and beneficiary OVCs and their parents/guardians and others. 5. Identify lessons learned and challenges and make clear, explicit and actionable￾recommendations, suggest options if the program is still relevant and desirable for future similar project design and implementation, that contributes to the attainment of the new USAID/Ethiopia Country Development Cooperation Strategy (CDCS), Development Objective 3(DO 3). USAID/Ethiopia CDCS stated its education DO3: Improved Learning Outcomes, IR 3: Increase achievement in basic education particularly in early grade reading. V. BACKGROUND Children in Ethiopia face many challenges during childhood. As a child’s age increases, the likelihood of the child living with both parents decreases. Only 65.2% of 10-14 year-olds and 52% of children 15-17 live with both parents (Demographic and Health Survey - EDHS, 2005). Lack of parental care and support due to HIV/AIDS exposes children to increasing vulnerability, such as food insecurity and chronic malnutrition, lack of protection/shelter, lack of access to education and physical and sexual abuse. These children also face the increased burden of caring for ill parents and of stigma and discrimination. This vulnerability can, in turn, increase children’s risk for contracting HIV/AIDS. According to the single point estimates for the year 2010 (FHAPCO, June 2007) there are 804,184 children orphaned due to HIV/AIDS in Ethiopia. There are over 25,000 primary schools in Ethiopia with a student population of over 17 million, and based on current anecdotal data, orphans and vulnerable children (OVC) constitute, on average, 12% of the total student body in primary schools. These children are more likely to suffer from malnutrition due to lack of food, lack of access to health care, experience stigma and discrimination, and prone to drop out of school. While access to quality education can only significantly reduce children’s risk of contracting HIV, schools also have the potential to provide a safe environment and often-missing adult guidance and supervision for children already affected by HIV/AIDS. Since many of the communities have access to schools in their area, the SCOPSO project was designed to utilize them as hubs through which communities can be mobilized to provide care and address the multitude of unmet needs of OVC. USAID/Ethiopia has nearly five years of experience “wrapping” its education and HIV/AIDS program by providing support to OVC at primary schools. The wraparound PEPFAR and education OVC program started in 2006 by providing scholarships for five thousand children in 100 schools in Amhara, SNNPR and Oromiya Regions. The OVC support integrates health and 43 education services in a seamless fashion to address the needs of the OVC holistically. It provides a menu of essential services based upon an initial mapping exercise to identify needs, gaps, and opportunities. Essential services include education, food and nutritional support, shelter and care, protection, health care needs of OVC, health care for HIV positive children and psychosocial services. The internal assessment of phase 1 (2006/07- 2007/08) implementation of education and HIV/AIDS wraparound project called Communities and Schools for Children Affected by HIV/AIDS (CASCAID) showed that after the intervention of the project, schools and communities placed the concerns of OVC as a higher priority as they now support OVC practically and systematically, creating a sense of responsibility in the minds of the community. The project has created a better hope and future for OVC by realizing that there are people concerned about them. Assessments made on the academic performance of the OVC had shown better rates of promotion for the OVC (84%) than the overall students’ rate of promotion (70%), and the dropout rates were also lower for the OVC (2%) compared with other students (10%). Moreover, 96.4% of the OVC in 2007/08 enrolled for the 2008/09 school Year.As a result of the first phase of education and HIV/AIDS wraparound program, USAID/Ethiopia has scaled-up this activity to reach 52,000 OVC in 400 primary schools across all regions of Ethiopia. The USAID/ Ethiopia School- Community Partnership Serving OVC affected and infected by HIV/AIDS (SCOPSO) started in August 2009 and ends in August 2013. This four-year project, aims to strengthen the ability of schools and communities to participate actively and owner the design, implementation and management of OVC support activities at schools in sustainable way. The overall objective of the project is to build the capacity of 400 primary schools to serve as focal points for OVC care and support to at least 52,000 HIV affected or infected OVC leading to increased enrolment, retention and academic performance. The two key results that contribute to the achievement of this objective are: Result 1: Quality and comprehensive services provision to at least 52,000 HIV￾affected or infected OVC, with a focus on increased enrolment, retention, and academic performance. In order to attain this result, the following services are provided to OVC at school. 1. Education: Teachers and Care coordinators have followed up on OVC class attendance and maintained communication with their parents/guardians in order to ensure that OVC in need of academic support actually attend tutorial sessions. All OVCs are provided with school kits including writing materials and uniforms. 2. Health Care: established linkages and relationships with health care service providers including health stations and Kebele Administrations. These efforts have resulted in obtaining health services and waivers for free medical services for needy OVC. All 400 Schools also provided health education; age appropriate HIV/AIDS prevention messages, personal as well as hygiene and environmental sanitation education. 3. Food and Nutrition: Community and school support for food took the form of collecting in kind food contributions such as cereals from farming communities, establishing relationships with private restaurants, using proceeds from IGAs to cover food costs for need OVC, and schools’ internal contributions made by individual teachers and students. 44 4. Psychosocial support and life skills training: The major psychosocial problems of OVC were identified and psychosocial support service was provided to OVC. 5. Shelter and Care: The shelter and care services include foster care, clothing, provision of shelter, renovation of old residential homes. 6. Economic Strengthening: Examples of activities include managing tea rooms in school compounds, establishing mini Shops, gardening, tree and fruit plantations, and animal fattening (sheep and oxen). Beneficiary OVC and their guardians have also been involved in IGA activities. 45 Result 2: Strengthen School-Community Partnership in 400 primary schools to enhance their capacity to plan and manage OVC support programs. Activities in support of this result including: 1. Strengthen capacity and facilitate the function of RSEBs, WEOs, KETBs, school communities, PTAs GACs, WHOs and Woreda local administration in supporting increased access to education for beneficiaries; 2. Support mobilization efforts of PTAs to mobilize community stakeholders that can support OVC; 3. Provide technical assistance to school leadership, OVC committees at schools including the GAC, and the PTA and the executives; 4. Develop selection criteria for targeting the OVC in accordance with OGAC guidelines; 5. Identification of the OVC is done through the appropriate PTA and other community committees as well as the Woreda, Kebele and traditional institutions; 6. Participatory development of the action plan for each school to implement the OVC support effectively with active involvement of CBOs, Idirs and all concerned PTA, GAC, Anti-AIDS clubs, School principals, teachers, Kebele administrators, community leaders, mentors and guardians; 7. Facilitate the development and implementation of anti-stigma and discrimination programs at the school, and at the community level in general; 8. Mentoring counselors (i.e. teachers, PTA members, parents/guardians) in counselling skills; 9. PTA and the school leadership trained in developing their skills in managing and organizing remedial/tutorial classes and handling and managing OVC documentation, in report writing skills, financial management, management of IGA and to stem all harmful traditions in the communities that put children at risk of HIV infection; 10. Provision of funds to support school communities to initiate low cost approaches for income generation scheme; and 11. Teachers trained in psychosocial and empowered by participating in extra-curricular activities to support OVC and anti HIV/AIDS clubs in schools. Community Mobilization Communities have also mobilized resources from several other sources using the service maps. The major contributors of local resources include school communities (students and teachers), churches and mosques, Idris, kebele and Woreda Administrations, as well as Regional HAPCO and NGOs. The School Core Groups (SCG) played a major role in mobilizing the wider community and key influential actors such as religious leaders, community elders and various Community-Based Organizations (CBO). As a result, communities have contributed in cash and in kind to leverage project funds for OVC care and support services. VI.SCOPE OF WORK 46 The evaluators are required to assess the outcomes of the objective, key results and the illustrative activities as stated above. Specific questions related to the objective and results of the project are indicated below in order of importance. 1. How effective is the project in achieving its set, objective and anticipated results? a) What the achievements against planned objective are as embedded in the M & E plan? b) Which services provided to OVCs are critical to improve the retention, enrolment and academic performance of OVCs? c) How effective is the project in linking the services provided to OVC to other sectors such as health and others? d) How effective is the project in addressing the needs of girl/female OVCs? e) How cost effective is the project in attaining its objective and results, particularly on cost per OVC? f) What has not been achieved and why? 2. How is the project’s approach and methodology designed to achieve the project objective and results? a) How effective is the collaboration, coordination and working relationship among the implementing partner and stakeholders such as school PTAs, local Government institutions, NGOs, CBOs and others? b) How effective is SCOPSO management structure and staff composition in terms of (i) relevant to the project intervention, (ii) Monitoring and Evaluation procedures and standards; and (iii) the overall project management environment? c) What institutional arrangement did implementing partner make to ensure sustainability of the project’s results? 3. What are the levels of satisfaction of the Ministry of Education, Regional Education Bureaus, Woreda Education Offices, schools, communities and beneficiary OVCs and their parents/ guardians? And why? 4. What are the lessons learned from the project? What are challenges encountered and measures taken? VII. METHODOLOGY Although the evaluation team is responsible to develop an appropriate methodology that answers evaluation questions, USAID/Ethiopia expects that both quantitative and qualitative methodologies will be employed. The quantitative methodology is to be used to collect data from secondary sources and selected beneficiaries (using appropriate sampling method). Using appropriate data collection tools, the evaluation team will undertake desk reviews (review of project and other relevant documents) and field visits to selected project operation areas. Key informant interviews at implementing partners’, stakeholders including direct beneficiaries and communities levels; focus group discussions and one-on-one interviews with selected beneficiaries and personal observation will be among the data collection techniques. The final methodology will be developed by the team based on the identified evaluation questions. 47 Team planning meeting (TPM): The assignment will commence with a half day Team Planning Meeting (TPM). This meeting will allow the team to meet with the USAID/Ethiopia BES staff to be briefed on the assignment. It will also allow USAID/Ethiopia to clarify to the team with the purpose, expectations, and agenda of the assignment. In addition, the team will clarify roles and responsibilities; review and develop final survey questions; review and finalize the assignment timeline and share with USAID/Ethiopia; develop data collection techniques, instruments, tools and guidelines; review and clarify any logistical and administrative procedures for the assignment; establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion; develop a preliminary draft outline of the team’s report; and assign responsibilities for the final report. Document and Literature Review: the evaluation team will review documents including but not limited to the Task Order Proposal, Annual and Quarterly project reports, GOE education strategies, studies and reports from the partner and from other countries with similar projects. Interviews and focus group discussion: The evaluation team will conduct key informant interview (school teachers, principals, PTAs, OVCs, parents/guardians) and focus group discussions with government partners at Regional, Woreda, school and community level. These include Education, Health, Women, Children and Youth and HAPCO bureaus//Offices, schools and others as deemed necessary. The team wills also carryout case studies/success stories. Site visit: The evaluation team will conduct visits to selected school under the project to observe and document how they are functioning and benefiting the OVC in schools and communities. From selected schools, the evaluation team will also select OVCs and their parents/guardians using appropriate sampling method for on-on-one interview. Sample Size: Once selected, the Contractor will prepare a detailed assessment framework including sample size and instruments which will be reviewed and approved by USAID/Ethiopia. The assessment work will include field visits to a representative sample drawn from the list of schools by Woreda and Region in Ethiopia. The sample regions will be Afar, Amhara, Harari, Oromia, SNNP, and DereDawa. From these regions, at least 20% of the target schools shall be included in the evaluation. The contractor can also propose sample of non-project schools for comparison. VIII. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT Team Composition: A five person evaluation team should be comprised of education and health experts. The Evaluation Team Leader will be responsible for team coordination, ensuring the timeliness and quality of deliverables. USAID/Ethiopia may propose internal staff members from the Missions to accompany the team during site visits or participate in key parts of the evaluation. Team Qualifications • Team Leader with Education Experience with particular expertise in education and health program/project evaluation, with 10-15years of practical experience in Ethiopia and /or other African countries and who has led at least two similar evaluation/studies. (Hired through the evaluation contractor); 48 • Two Education Evaluation specialists with 8-10 years of experience in managing and evaluating education and health integrated projects in Ethiopia and/or other African countries. (Hired through the evaluation contractor); • Two Health specialist with 8-10 years of experience in community based health and HIV/AIDS programing, management and evaluation in Ethiopia and/or other African countries. (Hired through the evaluation contractor). The consultant can hire additional support staff, in particular for one-on-one interview of beneficiaries but staffing arrangement will be evaluated with above sited key members of the team. An illustrative table for Level of Effort (LOE)—Dates may be modified based on availability of consultants, key stakeholders and time for field work. Activity Team Member(s) Estimated total number of days Total LOE Period of Performance (illustrative depending on start date) Selection of Evaluation Contractor USAID/Ethiopia - February Review of documents and begin drafting evaluation protocol and survey instruments; logistics 5 Evaluation Team 4 days each 20 March In-briefing with USAID, team planning meetings; finalize work plan, protocol, and survey tools; organize logistics for field work 5 Evaluation Team 3 days each 15 March Fieldwork including travel days 5 Evaluation Team 23 days each 115 March Preliminary data analysis and synthesis; drafting report and presentation materials 5 Evaluation Team 6 days each 30 April Debriefing of Mission staff—draft report submitted 5 Evaluation Team 1 days each 5 April Mission sends technical feedback/comments on the draft report to the team leader USAID/Ethiopia - - April Draft revised by the team leader and the evaluation contractor submits final report to Mission 5 Evaluation Team 5 days each 25 April Missions approves report USAID/Ethiopia - May Total number of days 50 250 - Travel over weekends may be required during site visits. IX.LOGISTICS The evaluation contractor will be responsible for all travels and logistics. 49 X. DELIVERABLES AND PRODUCTS Based on the above stated purpose, objectives, and key tasks, the evaluation team will submit the following deliverables: Evaluation framework: Including revised evaluation questions, detailed approach/methodology, survey protocol, data collection tools, and plans for analysis and dissemination of findings. The Team Leader will submit the evaluation framework to USAID/Ethiopia. USAID/Ethiopia will then review the proposed work plan/methodology and data collection tools and submit comments to the Team Leader prior to field work. The evaluation team will revise the work plan/methodology and data collection tools and send the final version to USAID/Ethiopia. The evaluation framework must be finalized and approved prior to the initiation of the interviews and site visits. Interim Briefings including status reports: The Team Leader will provide weekly status report on the evaluation plan implementation to USAID/Ethiopia. PowerPoint Presentation (in MS PowerPoint) used during debriefing to USAID/Ethiopia staff on the preliminary findings and recommendations that address set of objectives and associated questions. Draft report in English no longer than thirty pages, excluding coversheets and appendix. The report shall follow the general format indicated below: i. Coversheet indicating type of evaluation ii. Table of Contents iii. Acknowledgments iv. Acronyms v. Executive summary (3-5 pages) vi. Introduction vii. Background viii. Scope and Methodology ix. Body of the Report x. Summary of findings xi. Lessons learned xii. Recommendations xiii. References xiv. Appendix (includes, but not limited to, SOW, data collection instruments,sources identified, and people contacted or interviewed, statements of differences “regarding significant unresolved differences of opinion by funders, implementer, and/or members of the evaluation team, if any). The findings and recommendations should address set of project objectives, anticipated results and evaluation questions. All findings and recommendations should be linked to data gathered and referenced in the evaluation report. The Team Leader will submit the first draft report to USAID/Ethiopia at the end of the evaluation team’s visit. The Mission will provide consolidated, written comments to the evaluation team within 5 working days of receiving the draft report. 50 Raw Data: The evaluation team will provide electronic files of all raw data to USAID/Ethiopia for future use and submission to a data warehouse. Final report: Will address the Mission’s comments. The Team Leader will submit the final unedited report to USAID/Ethiopia within 5 working days after the team receives consolidated comments from USAID/Ethiopia. The evaluation contractor will provide the edited and formatted final document approximately 5 days after USAID/Ethiopia provides final approval of the content. Procurement sensitive information will be removed from the final report and incorporated into an internal USAID Memo. The remaining report will then be released as a public document on the USAID Development Experience Clearinghouse (DEC)(http://dec.usaid.gov)and the evaluation contractor’s web site. The Contractor shall submit edited and formatted final document in 5 hard copies and electronically. XI.RELATIONSHIPS AND RESPONSIBILITIES The evaluation contractor will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: • Recruit and hire the evaluation team. • Make logistical arrangements for the consultants, including travel and transportation, in￾country travel, lodging, communications and others. The USAID/Ethiopia BES Office will provide overall technical leadership and direction for the evaluation team throughout the assignment and will undertake the following specific roles and responsibilities: • Respond to any queries about the SOW and/or the assignment at large. • Consult Conflict of Interest: To avoid conflicts of interest or the appearance of a COI, review the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractor or NGOs evaluated/assessed and information regarding their affiliates. • Documents: Identify and prioritize background materials for the consultants and provide them, preferably in electronic form. • Site Visit Preparation: Provide a list of schools by Region and Woreda for site visit locations, and key contacts. • Mission Point of Contact: Throughout the evaluation work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. • Facilitate Contact with Implementing Partner: Introduce the evaluation team to implementing partner and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. • Timely Reviews: Provide timely review of draft/final reports and approval of the deliverables. XII. INSTRCUTION FOR THE PREPARATION OF THE PROPOSAL Technical Proposal 51 Bidders shall prepare a technical proposal consisting of three factors below: Factor 1 – Technical Approach(7 page maximum) Technical approach at a minimum shall include the following information: 1. Description of the contactors proposed methodology to complete the evaluation as described in the SOW. 2. Draft work plan to include: a. Proposed timeline; b. Proposed overall evaluation schedule; c. Deliverables; and d. Draft outline of the evaluation report. Factor 2 – Staffing Plan(2 page maximum) The staffing plan at a minimum shall include the following information: 1. Description of the management structure of the proposed team. 2. Delineation of the roles and responsibilities of the members of the evaluation team to ensure coverage of all elements of the statement of work. 3. Description of the ability to access skilled staff to complete the evaluation 4. Resumes of members of the evaluation team (Not included in the 2 page limit – No page limit is established for submission of Resumes.) 5. Confirmation of the availability of the evaluation team throughout the completion of the evaluation. Factor 3 – Past Performance Information(2 page maximum) Bidders briefly describe their past performance on similar projects. Past performance information shall include the following: 1. Up to three of the most recent and relevant contracts for efforts similar to the work detailed in the SOW. 2. For each contracts requested above, list contact names, job titles, mailing addresses, phone numbers, e-mail addresses, and a brief description of the work performed to include: • Scope of work or complexity/diversity of tasks; • Primary location(s) of work; • Term of performance; • Skills/expertise required; • Dollar value; and • Contract type (i.e., fixed-price, cost reimbursement, etc.) Price Proposal A firm bidding on this activity (Evaluation) must, in addition to its technical proposal, submit budge (in Excel) showing the projected level of effort (LOE) for each proposed full time and/or short￾time member of the team including subject matter expertise and logistical support. All other costs such as travel and per diem, in country costs for data collection and interviewing, communication, report preparation and reproduction should be included as appropriate. XIII. EVALUATION CRITERIA/FACTORS 52 For overall evaluation purpose, technical factors are considered significantly more important than cost/price factors. The technical criteria are presented below in descending order of importance. There are no sub-criteria. The bullet statements listed under each technical criterion are illustrative, but not exhaustive, of the considerations that make up that criterion. These criteria have been tailored to the requirements of this assignment and serve to (a) identify the significant matters that the bidders should address in their proposals and (b) set standards against which all bidders will be evaluated. Technical Proposals Evaluation Criteria The specific evaluation criteria and corresponding weights are as follows: Technical Evaluation Criteria Weight Factor 1 - Technical Approach 45 points Factor 2 - Staffing Plan 40 points Factor 3 - Past Performance 15 points Total Possible Evaluation Points 100 Technical Approach The Technical Approach shall be evaluated in accordance with the following: • Demonstration of a sound technical approach to complete the work outlined in the SOW, including the bidder’s approach to conducting evaluation. • Demonstration of an effective draft work that meets the requirements of the SOW and provides for realistic timelines, deliverables, and an effective draft outline of the final evaluation report. Staffing Plan The staffing plan shall be evaluated in accordance with following: • Demonstration of a sound and effective management structure of the proposed team, including clear expectations of roles and responsibilities of the members of the evaluation team to ensure coverage of all elements of the statement of work. • The demonstration of professional qualifications, education and relevant experience of its proposed personnel, particularly in conducting evaluations of projects/programs working on education development, health, HIV/AIDS and OVC in Ethiopia and other African countries. Past Performance Past performance shall be evaluated based on the implementation of projects or engagements similar in scope, size and complexity as evidenced by performance records and the testimony of clients. Bidders without evidence or record of relevant past performance will be evaluated neutrally for this criterion. The bullet statements listed below are illustrative considerations that make up this criterion. • Exhibits past record of quality service provision, including consistency in meeting goals and targets. • Exhibits a past record of timeliness of performance, including adherence to contract schedules and other time-sensitive project conditions, and effectiveness of team management to make prompt decisions and ensure efficient completion of tasks. • Exhibits records of customer satisfaction, including satisfactory business relations, addressing the history of professional behavior and overall business-like concern for the 53 interests of the customer, cooperative attitude in remedying problems, and timely completion of all administrative requirements. • Exhibits records of effectiveness of team management, including appropriateness of personnel for the job and prompt and satisfactory changes in personnel when problems with clients where identified. USAID/Ethiopia reserves the right to verify the experience and past performance record of cited projects or other recent projects by reviewing Contractor Performance Reports (CPR’s), other performance reports, or to interview cited references or other persons knowledgeable of the bidder’s performance on a particular project. The Government may check any or all cited references to verify supplied information and/or to assess reference satisfaction with performance. References may be asked to comment on items such as Quality of Product or Service, Cost Control, Timeliness of Performance, Customer Satisfaction, Key Personnel, and Utilization of Small Businesses. Bidders will be provided an opportunity to explain circumstances surrounding less than satisfactory performance reports if not previously provided the opportunity. USAID may also check other sources of information about the bidder including, but not limited to, other government agencies, better business bureaus, published media, and electronic data bases. 54 ANNEX VII: LIST OF INSTITUTIONS AND PERSONS CONTACTED Region Woreda Organization/institution Position of contact person/s Remark XIV. MISSION CONTACT PERSONs Allyson Wainer Office Chief, USAID/Ethiopia, BES Office Awainer@usaid.gov BefekaduGebretsadik Project Management Specialist USAID/Ethiopia, BES Office Bgebretsadik@usaid.gov AbdissaWoldeyohannes Acquisition and Assistance Management Specialist Office of Acquisition and Assistance (OAA) USAID/Ethiopia Awoldeyohannes@usaid.gov Awoke Tilahun Mission Monitoring and Evaluation Specialist atilahun@usaid.gov XV. REFERENCES (Project Document shall be available to the team once selected) • Task Order—Project Description • Annual Reports and most recent Quarterly Reports • Monitoring and Evaluation Plan and achievement toward targets • GOE Education Sector relevant documents and reports 55 ANNEX II: DATA COLLECTION INSTRUMENTS Tool 001 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project FOCUS GROUP DISCUSSION GUIDE FOR GUARDIANS OF OVC AND THE OVC (AGED 12 -17) IN THE SAMPLE SCHOOLS Dear moderator: Read (translate) the following to the participants: Welcome everyone, and thank you for joining us today. My name is __________ and this is ___________. We are working with PRIN International Research and Consultancy Services PLC. We are working with the USAID/WL Ethiopia HIV/AIDS Project. The USAID/WL Ethiopia Partner is one of the international organizations working on HIV/AIDS prevention and control following the existing national policy framework. In Ethiopia, USAID/WL is implementing HIV/AIDS program for the past many years mainly on provision of comprehensive care and support package to Orphans and Vulnerable Children (OVC). USAID/WL Ethiopia works in more than 8 regions in collaboration with schools and school communities. The project has supported OVC through health care, food and nutrition, education and psychosocial support. The supplemental activities also include support through legal protection, shelter and housing, nutrition and food, and economic strengthening. This focus group discussion (FGD) guide is intended for guardians of OVC/older OVC who are supported by the USAID/WL Ethiopia Partner. The purpose of this FGD is to gather information from you about the USAID/WL Ethiopia Partner project and its activities in Care and Support to OVCs and caretakers. This discussion will focus on three major areas: 1) how the project has been implemented, 2) what things hinder or help the implementation of the project, and 3) your recommendations for improving the project. Informed Consent Our discussion will be in the form of a conversation, and I hope you will talk to one another rather than just to us. We will ask you some questions which we have prepared. __________ will take notes, and we will also use a tape recorder so that we can have a record of what we discussed. Please feel free to answer the questions openly. You are the experts on life in this community and we want to learn from you. If you do not want to answer a question, you do not have to. We simply want to know your thoughts on the subject. All your answers will be kept confidential and your name will not be identified with the information you provide. Do all of you agree to participate in this group discussion? Yes _________ No _________ Now that we have introduced ourselves, would each one of you please tell us your name? Thank you. Now I will begin the questions. Moderator's Signature ________________ Moderator’s name: __________________________ Co-moderator’s name: _______________ Region: _______________________ Zone: _______________________ Woreda: ________________ Place: ________________________ Time:_______________________ Date: _______________________________ List of Participants 56 Part One 1. You know that USAID/WL Ethiopia Partner is implementing a project in this area. Let us discuss the types of activities implemented by USAID/WL Ethiopia Partner in your area? [Moderator: Please replace the USAID/WL Ethiopia Partner by the implementing partner/school in the area]. 2. What are the USAID/WL Ethiopia Partner's activities in your area with regard to OVC support? 3. What are the supports you have received from the program? Follow up questions: a. Food and nutrition b. sanitary material support c. Clinical care/Medical refund d. Provision of shelter/ housing refund e. Psychosocial support/spiritual support f. Economic strengthening/IGA g. Educational support (ask about attitude to learning and preparation/motivation of the OVC, their academic performance – improvement, as well as absence and dropout situation. Ask also what materials have been made available by the project to support the learning of these children.) h. Prevention activities I. Food and Nutrition, and sanitary material support Probe for: a. Adequacy and provision of food and nutrition to OVC by USAID/WL Ethiopia Partner b. Frequency and consistency of food/nutrition distribution c. Referral system of OVC to the care and support organizations such as WFP d. Screening of OVC based on nutritional assessment e. Provision of nutrition education f. Supply of sanitary materials (probe for type, quantity, & frequency) II. Clinical care/Medical refund Probe for: a. From where do OVC most frequently seek medical treatment? b. What prevents some OVC from accessing health services? c. Frequency and consistency of visits by home-based care volunteers Participant Number Sex Age(yrs) Marital Status (if guardian) Relation to the OVC (if guardian) P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 57 d. Free medical care and linkage of government waiver fee system e. Provision of free medical care service by private clinics f. Refunding of medical expenses by USAID/WL Ethiopia Partner and frequency g. Training on hygiene and sanitation h. Provision of sanitary materials such as soap, washing powder, bleach and sanitary pads III. Shelter and housing Probe for: a. Renovation of old houses and shelters by USAID/WL Ethiopia Partner b. Refunding for house renting c. Access to kebele houses IV. Psychosocial support/spiritual support Probe for: a. Frequency of visits by community-based counselors b. Counseling and spiritual support c. Spiritual counseling by religious leaders d. Participation in care and support clubs for guardians and care givers e. Visits by trained community volunteers and professionals V. Economic strengthening Probe for: a. Participation in income generating activities by USAID/WL Ethiopia Partner (both guardians and older OVC), b. Savings c. Provision of trainings and seed money d. Access to microfinance’s VI. Educational support Probe for: a. Where do OVC attend school? b. What prevents OVC from attending school? c. What help is available to keep OVC in school? Probe: a. NGOs b. Government c. Community d. Provision and frequency of education materials, covering of school fees, provision of school uniforms, and tutorial programs by USAID/WL Ethiopia Partner e. Attendance and classroom participation f. Academic performance status g. Participation in vocational skill training h. Provision of educational counseling for OVC i. Participation in job counseling a. VII. HIV Prevention activities Probe for: a. Participation in school based prevention activities b. Participation in community sensitization programs c. Participation in community conversation d. Role of PTAs and School community in care and support for OVC 58 4. Which services provided to you or your children are critical to improve school enrollment, retention and school performance of OVCs? (Probe for all three indicators) 5. What is your overall impression of the USAID/WL Ethiopia Partner project? 6. What are the things you like or dislike about the USAID/WL Ethiopia Partner project? (Probe why like or dislike?) 7. What have you gained in this USAID/WL Ethiopia Partner project? [Probe: in terms of psychological, financial, material, and other benefits] 8. What should have been done better in the provision of the aforementioned services? And how? (Probe for each service) Part Two 9. What are the major strengths of USAID/WL Ethiopia Partner project in this area? 10. What are the major challenges of USAID/WL Ethiopia Partner in this area? Part Three 11. What are your recommendations for improving the USAID/WL Ethiopia Partner project activities? This is end of our discussion, thank you very much for your time You can ask me any question you want to discuss 59 Tool 002 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project IN-DEPTH INTERVIEW GUIDE FOR IMPLEMENTING PARTNER’S (WORLD LEARNING AND SCHOOLS) STAFF Interviewer’s name: _________________________ Region___________Zone___________Woreda_______ Respondent Type/Detail_____________________ Date: _______________________________ Place: __________________ Category____________________________ Dear Interviewer: Read (translate) the following to the participants: Welcome everyone, and thank you for joining us today. My name is __________ and I came from PRIN International Research and Consultancy. We are working with the USAID/WL Ethiopia HIV/AIDS Project. As you might be informed, USAID/WL is conducting an end-line evaluation of SCOPSO Project. As your organization is implementing one of the USAID/WL supported HIV/AIDS projects, I am here to document the promising practices and lessons learned in your project. I hope you would understand the contribution this assessment will have to the improvement of your project in particular and to a better design of such programs in general. This In-depth InterviewGuide is developed to collect information from Implementing Partner’s staff (School core workgroup members, teacher coordinatorsof core services, homeroom teachers, principals, Project Officers from WL) who oversee the implementation of the USAID/WL supported HIV/AIDS Prevention, Care and Support Project. Informed Consent I want you to know that if you decide to participate in this interview and answer questions, what you tell me will be kept private. All of the answers we get from various interviews will be put together so that no one will know what you told me. Because your answers are important, I would like to ask you to give me your true responses. Do you agree to talk with me and answer some questions? Yes _________ No _________ Explain to participant that you want to tape the discussion so that you do not have to remember everything. With his/her consent, turn on the tape recorder. Ask participant if he or she has any questions about the process, the study, or the consent prior to starting the interview. Signature___________________ Part one – General Information 1. Please tell me when you started working for this project? In what capacity are you working for this project? 2. Can you tell me your specific duties and responsibilities? 3. Could you tell me a little bit about your project? How was SCOPSO initiated? Who initiated it? When was it started? Its life span and goals and objectives? Did you participate in designing process? 4. Please tell me more about the project? Was it relevant to the context of your community or OVC in school (The number and types of beneficiaries, target areas, CBOs involved-if any) 5. What are the specific components of the project? 6. Who are your key stakeholders in the implementation and coordination of this project? 60 Part two – Project Implementation 7. What kind of specific services have you been providing in each sub-component of the project? Who are the target groups for each service? And how many are they? 8. What does the project implementation structure look like? (Probe for people and agents involved at the coordinating office, branch offices and service delivery points to the level of household/community groups) 9. In your opinion, are there adequate workers and volunteers to implement project activities? 10. How does the coordination of these different structures (and people involved) within the project look like? What specific achievement can you tell in this regard? What about the challenges? 11. Are all project components being implemented effectively? Which specific project services/components are effective compared to the others? a. Does the program use community based strategies to increase OVC and their guardians’ access to food (e.g. school feeding programs, community gardens, seeds, community kitchens)? b. Is the program mobilizing community resources (including labor, materials) to improve shelter and care for OVC in the community? c. Does program link caregivers to appropriate IGA based on market demand, interest and skill level of caregiver? d. Is the program linking with appropriate child protection bodies for legal protection of children? e. Does the program facilitate free access to health services for OVC and guardians? f. Is the organization implementing programs to raise community members’ awareness including caregivers of psychosocial support needs for OVC and their families? g. Is the program identifying and promoting educational opportunities for OVC? 12. Can you tell me the specific reasons for project activities/components which are implemented effectively? What about the reasons which hinder effective implementation of the other components? 13. To what extent did you build the capacity of local partners? [Probe to specify the type of capacity building activities undertaken] 14. How do you provide administrative and technical management, including direction to your local partners? 15. What are the most prominent project achievements? What has contributed to the achievement of these activities/services? 16. What are the major challenges that made implementation of the project difficult? What [Probe how they have tried to address these challenges in the past] 17. What are the most frequently mentioned services/activities or approaches for which your beneficiaries are satisfied? What do you think are the reasons that made beneficiaries stratified with these services? 18. What are the most frequently reported complaints from project beneficiaries? What do you think are the reasons for such complaints? 19. What kind of specific support have you got from the stakeholders mentioned above? 20. Please tell me your system of monitoring, evaluation and reporting of program activities. [Probe: Whether the system is prepared by the head office or branch offices?] 61 21. How do you use information gathered from monitoring activities to improve interventions for OVC? 22. Do you have annual, quarterly, or monthly plan at office level? Who prepared that and how do you relate it with project main plan? 23. To what extent is the project operating the way it was intended to work originally? Is there any change to the original plan of the project? If yes, why? 24. Do you have any sustainability plan? Please tell me if you have one? 25. What exit strategy is planned for termination of the project? Have you received one or you prepared by yourself. Part three – Promising/Best Practices in the Project 26. Are there specific promising/best practices that you can share to me? [Probe for specific services as mentioned below] - Educational Support (ask teachers also about attitude to learning and preparation/motivation of the OVC, their academic performance – improvement, as well as absence and dropout situation) - Health Care - Shelter and Care - Food and Nutrition - Psychosocial Support - Economic Strengthening - Coordination and Networking - Monitoring and Evaluation 27. What are your criteria to mention these activities as promising practices/best practices? 28. Do you have documented best practices that you can share to me? Part four – Lessons learned in the implementation of the project 29. What have you learned from the major challenges that you faced in the implementation of this project? [Probe by specific services as mentioned below] 30. What have you learned from the major achievements of this project? [Probe by specific services as mentioned below] 31. What do you think would make the project better? 32. Please tell me your recommendation on the way forward? If we were to improve or sustain the program? 33. What do you think as a future program strategy and an implementation structure do you want to look like? 34. Is there anything else that you want to share with me that we have not talked about today? _____________________________________________________ Thank the respondent! 62 Tool 003 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project IN-DEPTH INTERVIEW GUIDE FOR THE PTA Interviewer’s name: _________________________ Region___________Zone___________Woreda_______ Respondent Type/Detail_____________________ Date: _______________________________ Place: __________________ Category____________________________ Dear Interviewer:Read (translate) the following to the participants: (Note: when you read the sheet, please replace USAID/WL with the name of local implementing partner ) Welcome everyone, and thank you for joining us today. My name is __________ and I came from PRIN International Research and Consultancy. We are working with the USAID/WL Ethiopia HIV/AIDS Project. As you might be informed, USAID/WL is conducting an end line evaluation of SCOPSO Project. As your organization is implementing one of the USAID/WL supported HIV/AIDS projects, I am here to document the promising practices and lessons learned in your project. I hope you would understand the contribution this assessment will have to the improvement of your project in particular and to a better design of such programs in general. This In-Depth Interview Guide is developed to collect information from the PTA about exisitng HIV Prevention and care programs achivements, lessons and promising practcies. This interview will focus on 3 main areas: 1) how the project has been implemented and role of PTAs on the process, 2) how is the situation of OVC living in the area, and 3) Challenges, lessons and promising practices and your recommendations for improving the project. Informed Consent I want you to know that if you decide to participate in this interview and answer questions, what you tell me will be kept private. All of the answers we get from various interviews will be put together so that no one will know what you told me. It is up to you to decide if you want to talk with me in this interview. If you decide to answer some questions today, you can still refuse to answer any questions you don’t want to answer, or stop the questions completely at any time. Because your answers are important, I would like to ask you to give me your true responses. Do you agree to talk with me and answer some questions? Yes _________ No _________ Explain to participant that you want to tape the discussion so that you do not have to remember everything. With his/her consent, turn on the tape recorder. Ask participant if he or she has any questions about the process, the study, or the consent prior to starting the interview. Part one – General information about the project and role of PTAs in the process, 1. What is the objective of this PTA? What capacity does the PTA members and/or officials have on OVCsupport? 2. What is the impact of HIV/AIDS in this community? On what evidence is your answer based? 3. How does your PTA perceive HIV/AIDS and people who are infected and affected by HIV/AIDS? 4. What is your involvement/participation in the USAID/WL-ET project? 5. What support did you get from the project? and for how long? 6. Was the support helpful to fulfill your demand? 7. Have you participated in any training activity of the project? If yes please explain? 8. Do you feel that you are capacitated and feel confident to perform by yourself if the support from the project is stopped completely?[Probe specifically to their involvement] 63 9. What changes have you observed to the life of program participants like you and your organization? Please tell me the changes and benefits specifically? Part Two – How is the situation of OVC living in the area, Education: 10. Are all school age children attending school? What prevents children from attending school? Prompt about OVC, Children from affected families? 11. What help is available to keep children in school? 12. How is the motivation of the children, their academic achievement, attendance as well as dropout situation? 13. What resources have been supplied to the schools in the name of the OVC? Are the resources being made use of? Health 14. When a child is ill does she/he go for medical treatment? What prevents some children from accessing health services? 15. Is there any difference between OVC and non-OVC in accessing health services? (Non OVC are the less vulnerable) Nutrition 16. What does a normal meal for a child consist of in this community and how many meals in a day? WHAT IMPACT HAS HIV/AIDS HAD ON THE AMOUNT OF FOOD THAT HOUSEHOLDS HAVE? 17. How do OVC households cope in providing food for all the children? 18. Are OVC receiving nutritional support? Describe Psychosocial Support: 19. What types of psycho-social support do the children in your area receive? (Probe counselling, recreational activities, spiritual, succession planning, etc) Legal Support/Protection 20. What happens to the property when the spouse/parents die? Is there a difference between when a man guardian dies and when a woman guardian dies? 21. What type of property is grabbed? How common is property grabbing? How many cases in the community in the last one year. 22. What community actions are there to prevent property grabbing and other right abuses? What makes these community actions effective or ineffective? 23. How do you describe the role of legal units in your area in protecting the rights of OVC? Shelter and Care/Basic Material Needs 24. Describe the overall housing status of OVC? 25. What do you think of major material needs of OVC? (excluding education, health care, and nutrition/food) 26. Is there any system in place to address theses gap? If yes, describe, if no why? Economic Strengthening 64 27. Is there any Income generating schemes for OVC that the PTA recognizes? What external support do you receive? 28. How do you describe/rate the success and challenges of ongoing IGA schemes? 29. Describe the role of your PTA in the ongoing IGA schemes? 33 Is there any referral linkages set in your area? How do you describe the existing collaboration between different stakeholders in HIV prevention, care and support program? Part Three – Challenges, Lessons and promising practices and recommendations 34 What do you generally feel about this project? Can you say it is very useful for beneficiaries and local communities? What are the main challenges you witnessed in the process of project implementation? 35 What do you think would make the USAID/WL-ET’s project better? Is there anything else that you want to share with us that we have not talked about today? 36 Please tell us the lessons you learned from the process? 37 Do you think your PTA is ready to take over the program at the end of the project time? This is end of our Interview, thank you very much for your time You can ask me any question you want to discuss 65 Tool 004 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project KEY INFORMANT INTERVIEW GUIDE FOR GOVERNMENT OFFCIALS /PARTNER LINE OFFICIES Interviewer’s name: _________________________ Region___________Zone___________Woreda_______ Respondent Type/Detail_____________________ Date: _______________________________ Place: __________________ Name of Institution represented by Respondent_______________________________________ Interviewer: This Key Informant InterviewGuide is developed to collect information from Government offcicials who oversee the implementation of the project activities. Welcome everyone, and thank you for joining us today. My name is __________ and I came from PRIN International Consultancy &Research Services PLC. We are working with the USAID/WL Ethiopia HIV/AIDS Project. As you might be informed, USAID/WL is conducting an end line evaluation of SCOPSO Project. As your organization is implementing one of the USAID/WL supported HIV/AIDS projects, I am here to document the promising practices and lessons learned in your project. I hope you would understand the contribution this assessment will have to the improvement of your project in particular and to a better design of such programs in general. The aim of this interview is to obtain information, lesson learnt and promising practices about USAID/WL￾ET’s HIV AIDS Prevention and care & Support Project and its activities. This interview will focus on four major areas: 1) project design and implementation;2) hindrances orfacilitators of the project;3) the best practices related to the project implementation; and 4) your recommendations for improving the project. Informed Consent I want you to know that if you decide to participate in this interview and answer questions, what you will tell me will be kept private. All of the answers we get from various interviews will be put together so that no one will know what you told me. Because your answers are important, I would like to ask you to give me your true responses. Do you agree to talk with me and answer some questions? Yes _________ No _________ Explain to participant that you want to tape the discussion so that you do not have to remember everything. With his/her consent, turn on the tape recorder. Ask participant if he or she has any questions about the process, the study, or the consent prior to starting the KII. Signature___________________ Part one – project design and implementation 1. Please tell me how much you know about this project? (Probe for objectives, activities of the project, etc.) 2. How are the project’s objectives relevant to the socio-economic context of the country and its main development challenges? (Probe: how SCOPSO contributes to quality of education improvement, health, well-being, etc.)Were they in line with the country’s stated national development priorities? 3. How do you see the project interventions in terms of addressing related problems in the community? in what ways? 4. How do you evaluate the project in relation to appropriateness and relevance to national policy priorities and addressing community and individual needs? What are the strength and weakness during implementation of the projects 5. Were the objectives and expected results in line with the beneficiaries’ interests? 6. Were national counterparts actively involved in the inception and design process of the project so as to enable national ownership in the implementation stage? 7. Please tell me about your role in this project? What is the level of your involvement in this project? 8. How is your working relationship with the project? Please tell me about the areas of collaboration and types or working relationship? [If no probe why the reason was and terminate.] 66 9. How do you rate the performance of this project with regard to attainment of its objectives? 10. If there are other organizations running similar project in your area please tell me the comparative strength and shortfalls of your (this) project Part two – facilitators and barriers to implementation 11. How do you see the progress of the project? (in terms of achieving planned objectives) 12. What do you think the reasons that promote or hinders to the implementation of the project? 13. Please tell me the major achievements the project registered in terms of changing the lives of the target OVCs. 14. What were the best things/activities you like about the project? Why? What are the success stories during implementation of the project? Can you give me examples? 15. What do you like to be improved on the project activities? Which part of the project? In what way? Part three – Recommendations and Way forward 16. Please tell me your recommendation on the way forward? If we were to improve or sustain the program? Part four – other 17. Is there anything else that you want to share with me that we have not talked about today? Thank the respondent! 67 Tool 005 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project CASE DEVLOPMENT GUIDE Dear Moderator, please identify two OVC aged 12-17 (probably with the help of two different homeroom teachers) and develop cases about each of them. 1. Please give us a brief background about yourself (name, place of birth, current family status, household conditions (Probe: Where the respondent was born and brought up, with whom was he/she living, who was taking care of him/ her, etc ) 2. Describe the situation you were in before you registered for the school based care and support program. Discuss your situation regarding a. Educational status, school performance, etc. b. Food (availability of food in the household, etc) c. Health and Medical Conditions (Access to health care services during illness, etc) d. Psychosocial conditions e. Housing, place of living, clothing f. Other situation 3. How did you join the care and support program? Who identified you? When did you join the program 4. What services have you received from the program? Discuss in details a. Educational Support b. Health Care c. Shelter and Care d. Legal Protection e. Food and Nutrition f. Psychosocial Support g. Economic Strengthening 5. How do you describe the change observed in your life since you joined the program (Current status of informant)? Discuss in details regarding a. Educational achievements b. Health status c. Availability of Food d. Emotional wellbeing e. Clothing and housing conditions (including place of living ) f. Household Economic conditions g. Other changes occurred due to the project 6. What is your view about your current living condition? (Probe: the respondent’s views on feeling of success or failure in life and reasons for the responses) 7. Please tell us what you feel about the future? What is your vision? 8. Any further point you want to share regarding the program and your life? 68 Tool 006 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project Format for Extracting School Statistics on Enrollment, Retention and School Achievement Name of the school __________________________________________________________ 1) Location of the school: rural ____ Urban _____ (use local criteria) 2) Total number of enrollment Year Male Female Remark 2003 E.C. 2004 E.C. 2005 E. C. 3) Dropout rate for the school by grade level Grade 2003 E.C. 2004 E.C. 2005 E.C. 1 Male Female 2 Male Female 3 Male Female 4 Male Female 5 Male Female 6 Male Female 7 Male Female 8 Male Female 4) Total number of OVC who are beneficiaries of the particular project Number of beneficiary OVC Male Female Total Initial Current (2005 E.C.) 5) Number of dropout OVC by grade level and year Grade 2003 E.C. 2004 E.C. 2005 E.C. 1 Male Female 2 Male Female 3 Male Female 4 Male Female 5 Male Female 6 Male 69 Female 7 Male Female 8 Male Female 6) Academic achievements of beneficiary OVC Classroom rank of the OVC Before joining the project After joining the project Proportion of OVC children who achieved the rank of 75% and above Proportion of OVC children who achieved rank of 50% - 75% Proportion of OVC children who achieved rank of below 50% 7) List of school facilities the project made available for use by the OVC a) Library references: _____________________________________________________ b) Laboratory tools & chemicals _____________________________________________ c) Pedagogical resource center materials _____________________________________ d) Play materials (balls, etc) ________________________________________________ e) Any other learning aids (specify) _________________________________________. 70 Tool 007 USAID/ETHIOPIA Performance Evaluation of the School Community Partnership Serving Orphan and Vulnerable Children affected by HIV/AIDS (SCOPSO) project Format for Extracting Performance Statistics from intervention schools & Project Offices Indicators Male Female Remark Input/process indicators 1. Number of school PTAs/OVC committees functioning in support of OVC 2. Number of OVC identified with appropriate criteria and registered for support Output indicators/service delivery 3. Number of OVC receiving health care services 4. Number of OVC receiving educational support 5. Number of OVC receiving vocational education/training 6. Number of OVC receiving educational support 7. Number of OVC receiving psychosocial support 8. Number of OVC receiving food and nutrition support 9. Number of OVC household receiving economic strengthening 10. Number of OVC supported with IGA activities Output indicators: system strengthening 11. Number of schools with functional IGA activities 12. Number of OVCs referred to community based care and support services 13. Number and type of initiatives taken by local government, community organizations and NGOs and individuals to support OVC 14. Number of care givers trained and functional Outcome indicators 15. Ratio of school attendance of orphans to non-attendants 16. More than 95% OVCs retained/persisted in school from year to year 17. More than 95% OVC in each target school promoted to the next grade by end of academic year 71 Tool: 008 Standard Checklist for School Activities Instruction The objective of this checklist is to see the school activities against the critical minimum activities as stated in the standard service guidelines for orphan and vulnerable children. The checklist consists of 48 critical activities for the six standard service deliveries. The data collector will complete the checklist by ticking a ‘Yes’ or ‘No’ answer for the implementation of each critical activity. Region ___________________________________ Woreda ____________________________________ Name of School_____________________________________ Date of Visit __________________________________ Name of School Director _____________________________________ Type of School (intervention/non-intervention): ______________________________ Name of Data Collector ___________________________________________________ No. Critical Minimum Activities* Yes No Comments 1. SHELTER AND CARE Desired Outcome: All OVC have shelter clothing, personal hygiene and adult caregiver which is the same as or similar to the non-OVC in community. 1.1 Regularly assess the needs of OVC for shelter. 1.2 Identify and mobilize community resources to construct, improve and renovate shelter for OVC. 1.3 Advocate for the provision of alternative options to housing children such as day care, temporary shelter, etc. 1.4 Link and advocate with stakeholders (legal services, Kebeles, others) 1.5 Regularly assess the shelter and care needs of OVC. 1.6 Ensure that an adult/foster caregiver visits the child at home and provides appropriate support. 1.7 Refer children without adequate support to other services including temporary shelter. 1.8 Sensitize community, line government offices and other stakeholders to monitor progress of the children (status of shelter and care) 1.9 Recruitment, training and assignment of an adult/foster care giver or adoptive parents for OVC based on consent from OVC and caregiver. Includes training and provision of continuous support to caregivers to provide PSS to OVC 2. ECONOMIC STRENGTHENING Desired Outcome: Households caring for OVC have additional and diversified source of income to care for family. 2.1 Assess household situation in which OVC live and determine if there is income to support needs of children. 2.2 Refer caregivers to IGA opportunities (savings groups, etc). 2.3 Map service providers and leverage resources from the private sector for training and future employment of those trained. 2.4 Conduct market analysis for business viability before training. 2.5 Help households caring for OVC to get financial resources. 2.6 Provide training on how to generate and manage income. 2.7 Provide materials, financial, and job opportunities. 2.8 Monitor/document progress of beneficiaries through an assessment checklist. 3. PSYCHOSOCIAL CARE AND SUPPORT Desired Outcome: OVC develop personal strengths and skills to become self-confident, happy, hopeful, and able to cope with life’s challenges. 3.1 Assess psychosocial needs of children. 3.2 Identify and address barriers to PSS for children. 3.3 Follow up regularly to monitor child’s status. 3.4 Mapping of PSS services including (child friendly centers, religious leaders). 3.5 Train volunteers in recognition of PSS needs and counselling 3.6 Provide re-integration services are provided for children who have lived outside of family care. 3.7 Establish support groups (children and guardian support groups and clubs) to counsel/support caregivers/children. 3.8 Develop safe spaces for children to engage in play. 72 3.9 Increase awareness amongst caregivers and community on parenting, positive disciplining, communication, open dialogue with children on RH and HIV and AIDS issues etc. 3.10 Provide life skills training through peer groups. 4. HEALTH SERVICES Desired Outcome: Child has access to health services, including HIV and AIDS prevention, care and treatment. 4.1 Assess and monitor the health status of OVC through household visits. 4.2 Refer OVC to health services based on need. 4.3 Follow up to ensure receipt of health services and identify whether child is better 4.4 Conduct mapping of health services in the community with participation of key stakeholders. 4.5 Ensure formal referral systems exist. 4.6 Provide basic age-appropriate health education and ensure that children receive HIV and AIDS education either directly from the CBO or through another partner, church, or community. 4.7 Train care givers/volunteers on a comprehensive range of health issues: hygiene, ART issues, IMAI, nutrition. 4.8 Make referrals for rape/child abuse/emotional problems, holistic care and follow up. 5. EDUCATION Desired Outcome: Child is enrolled, regularly attends, and completes a minimum of primary school (grade 8). 5.1 Regularly assess educational needs of OVC (enrollment, retention, promotion). 5.2 Identify and address barriers to education on an individualized basis for each OVC in collaboration with key stakeholders. 5.3 Conduct resource mapping for educational services. 5.4 Refer OVC to educational resources for tutoring, school materials (uniforms, etc). 5.5 Regularly follow up on children’s status. 5.6 Identify and engage all stakeholders, including Kebele Education and Training Board, PTAs and CBOs, etc. 5.7 Build capacity to support OVC among PTA, teachers, community representatives and local government officials. 5.8 Support life skills and livelihood opportunities as an integral part of the education program. 6. FOOD AND NUTRITION Desired Outcome: Adequate food is available for the child to eat regularly throughout the year for healthy and active life. 6.1 Assess food and nutritional needs of children. 6.2 Refer malnourished or food insecure children and families to food sources. 6.3 Follow up to ensure that children have received food or other rehabilitative/therapeutic service and monitor their status. 6.4 Identify (through mapping) and engage other stakeholders to strengthen linkages and referral systems for food. 6.5 Encouraging exclusive breast feeding and safe complementary feeding practices. * Source: Standard Service Delivery Guidelines for Orphans and Vulnerable Children’s Care and Support Programs, MOWA and FHAPCO, 2010 73 ANNEX III: DOCUMENTS REVIEWED • SCOPSO Contract Award No. 663-C-00-09-00422-00, USAID,2009. • USAID/SCOPSO Performance Plan, World Learning-Ethiopia, October 2009. • Institutionalization of Sustainable Care and Support for OVC at Primary Schools: Implementation Guideline, WL-E, June 2011. • USAID/SCOPSO Baseline Study Summary Report, WL-E, 2010. • USAID/SCOPSO Annual Progress Report for October 1/2009- Sept. 30/2010, WL-E, 2010. • USAID/SCOPSO Annual Progress Report for October 1/2010- Sept. 30/2011, WL-E, 2011. • USAID/SCOPSO Annual Progress Report for October 1/2011- Sept. 30/2012, WL-E, 2012. • Child Social and Financial Education: Reference materials for teachers, Vol. I & II, WL-E, 2013. • Psycho-Social Support at School, WL-E, n.d. • Other training materials of the project and resource documents 74 ANNEX IV: DISCLOSURE OF ANY CONFLICTS OF INTEREST 75 76 77 78 79 80 81 ANNEX V: ACHIEVEMENTS AGAINST THE SELECTED INDICATORS Performance Indicator Baseline value Life of Project Target Achievements Year 1 Year 2 Year 3 Result 1: Provision of quality, comprehensive services to at least 40,000 HIV-affected or infected OVC, with a focus on increased enrolment, retention, and academic performance 1. # of care givers trained (disaggregated by sex) - - 1002 (M=596, F=406) 1,894 (M=1,306, F=588) 1,435 (M=952, F=483) 2. # of OVC selected as the PEPFAR criteria 52,000 52,000 40,000 12,000 N/A 3. # of beneficiary OVC dropped out of schools - - 1,523 2,492 2,866 4. # of OVC received education supplies and materials - 52,000 40,000 52,000 52,000 5. # of OVC received tutorial support - - 29,266 45,833 52,000 6. # of successful OVC referrals to local service providers - 3,500 2,017 6,607 4,148 7. # of teachers trained in psychological support (PSS) provision - 840 1,002 N/A N/A 8. # of trained teachers providing PSS to OVC - 840 1,002 1,002 1,002 9. # of OVC received PSS/counselling service - 52,000 31,379 52,000 52,000 10. # of teachers trained in life skills education - 840 783 N/A N/A 11. # of OVC participated in life skills education - 26,000 13,295 15,257 5,199 12. # of OVC received food and supplemental nutrition support - 5,200 3,680 7,169 10,647 13. # of OVC received health service - 52,000 34,008 52,000 52,000 14. # of OVC received shelter and care service - 13,000 3,669 29,379 27,994 15. # of school based IGA programs - 400 168 400 400 16. # of OVC benefited from the school based IGA program - 13,000 - 16,960 25,562 17. # of guardians/older OVC supported from school based IGA program - - 470 4,457 11,625 Result 2: Strengthen school-community Partnership in 400 primary schools to enhance their capacity to plan and manage OVC support programs 1. % of OVC who sought and did not get medical treatment the last time they were sick during the year 48% 10% - 24.9% 24.9% 2. % of OVC with correct comprehensive knowledge of HIV and AIDS transmission and prevention 3% 55% - 11.2% 20.2% 3. % of OVC receiving care from a responsible adult 74% 90% - 64.3% 89.8% 4. % OVC who report less than two meals a day during the week before the survey 34% 15% - 35.9% 22.9% 5. % of OVC reporting that they discussed education issues with caregivers more than six times in a year 64% 95% - 76% 81.6% 6. % of Deviations between mean scores of OVC and non-OVC in all subjects 3.04 0.00 - 3.13 3.86 7. % OVC in grade 4-7 promoted to the next grade by end of the academic year (disaggregated by sex and grade) 91% 97% - 86% 86% 82 ANNEX VI: COVERAGE OF THE PROJECT AGAINST CORE SERVICES Indicators Male Female Total 18. Number of OVC identified with appropriate criteria and registered for support (Beneficiaries drawn from the sample schools) 2374 2733 5107 Proportion of OVC beneficiaries receiving: 19. educational support 100 100 100 20. psychosocial support 74.5 75.0 74.8 21. health care services 61.4 64.5 63.1 22. food and nutrition support 40.8 40.8 40.8 23. vocational education/training 19.8 21.1 20.5 24. economic strengthening 11.1 22.5 17.2 25. IGA activities 11.6 15.9 13.9 26. referred to community based care and support services 8.0 8.0 8.0 Note: The number of beneficiaries were taken from the sample schools’ statistics. 83 ANNEX VII: INSTITUTIONS AND INDIVIDUALS CONTACTED8 Region Zone/ Woreda Name of Institution/orga nization/School Contact personnel9 Remark Addis Ababa World Learning/ Ethiopia Chief of Party for SCOPSO project In-depth Interview Deputy Chief of Party for SCOPSO In-depth Interview M&E Officer for SCOPSO In-depth Interview Afar Summera World Learning/E Afar Area Coordination Office Area Coordinator In-depth Interview Amibara Woreda Education Office HR Unit Head In-depth Interview Ariba Trap Primary School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants One male student and one female student Case study Kusira Primary (non-intervention school) School principal Key informant interview Amhara Bahir Dar Amhara Education Bureau Planning & Resource Mobilization Process Owner Telephone Interview South Wollo/ North Wollo Wuachale Primary Kelala Primary Firfira Primary Liguama Primary Mersa Primary Woldiya Primary Taitu Bitul Primary Gobye Primary Hara Primary Wurgessa Primary School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants One male student and one female student Case study Yeju Genet (non￾intervention school) School principal Key informant interview World Learning/E Area Office Area Coordinator In-depth Interview Dessie Town Education Office Woreda Education Officer In-depth Interview WoldiyaTown Education Office Woreda Education officer In-depth Interview 8 An attempt to interview two directors from MOE was not successful because of their busy schedule at the time of data collection. 9 The identities of individuals contacted have not been specified for ethical reasons. 84 North Showa/ Oromia zone World Learning/E Area Office Area Coordinator In-depth Interview Zerai Yakob Prim. Model Kutir 2 Prim. Bruh Tesfa Primary Sela Dingay Primary Abeye Primary Senbete Primary Sedassa 29 Primary Kemise Primary Bati Kutir 1 Prim. Bati Kutir 2 Prim. School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Two OVC male students Case study Erqe (non￾intervention school) School principal Key informant interview Dire Dawa/ Harari Dire Dawa City Harer City Dire Dawa City Administration Education Bureau Planning & Resource Mobilization Officer Key Informant Interview Harari Regional Education Bureau Planning & Resource Mobilization Officer Key Informant Interview Dire Dawa City Woreda Education Office Education Coordinator Key informant interview Harar city woreda education office Education Coordinator Key informant interview World Learning/E Area Coordination Office Area Coordinator Key informant interview Kutir 1 Primary Aba Yohannes Prim Yeshimebet Prim. Aboker Primary School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants One OVC male student and three OVC female students Case study Sabian Kutir 2 School Principal Key informant interview Oromia Chirkos Oromia Education Bureau Planning & Resource Mobilization Process, SCOPSO focal person In-depth Interview Jimma World Learning/E Area Coord. Office Area Coordinator In-depth Interview Jimma Town Education Office Woreda Education Supervisor In-depth Interview Hermata Primary Jiren No. 1 Primary Seto Yudo primary School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants 85 Male and female OVC Students FGD Discussants 2 OVC beneficiaries Case study Yebu Primary (non￾intervention school) School Principal Key informant interview Bedelle Engile No.1 Janmeda Bedele primary (addisu) Lemlem Karl Ras Tesemma School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants 2 OVC beneficiaries Case study Bedelle Town Education Office Woreda Education Supervisor In-depth Interview Metu Abune Petros Prim. School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants 2 OVC beneficiaries Case study MetuTown Education Office Woreda Education Supervisor In-depth Interview Gore Gore No.1 Prim. Kundi Primary School principals In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants 2 OVC beneficiaries Case study SNNPR Alaba Kulito Denebe Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Kulito Primary School School principal and Deputy Principals (3) In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Alaba town Education Office Education Quality Improvement Officer and Supervisor Key Informant Durame Adelo Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary, PTA and Cluster Supervisor In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants 86 Ajoro Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Mierab Abaya Birbire Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Mollae Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Woreda Education office Office Head Key Informant Dalbo Primary (non-intervention school) School Principal Key informant Arba Minch town Gero primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Kulfo Primary School School principal In-depth Interview School SCG Committee members Group Interview School SCG secretary and PTA In-depth interview OVC Guardians FGD Discussants Male and female OVC Students FGD Discussants Arba Minch Education Office Arba Minch World Learning Arba Minch area Coordination Office Area Coordinator In-depth Interview Hawassa World Learning Regional Office Regional Coordinator In-depth Interview 87 ANNEX VIII: SAMPLE SCHOOLS BY REGION AND INTERVENTION STATUS Region Intervention Status Name of Primary School Amhara SCOPSO schools Wuachale, Kelala, Firfira, Liguama, Mersa, Woldiya, Taitu Bitul, Gobye, Hara, Wurgessa, Zerai Yakob, Model Kutir 2, Bruh Tesfa, Sela Dingay, Abeye, Senbete, Sedassa 29, Kemise, Bati Kutir 1, & Bati Kutir 2 Non-SCOPSO schools Yeju Genet, Erqe Afar SCOPSO schools Ariba Trap Primary school Non-SCOPSO schools Kusira Primary School Oromia SCOPSO schools Hermata, Jire Kutir 1, Ras Desta, Jidda, Ras Tesema, Lemlem Karl, Jan Meda, Bedele Primary (Addisu), Seto Ydo, Gore Kutir 1, Kundi, Engili Kutir 1, Abune Petros Non-SCOPSO schools Yebu Primary Harari Region & Dire Dawa City Administration SCOPSO schools Kutir 1, Aba Yohannes, Yeshimebet, Aboker Non-SCOPSO schools Sabian Kutir 2 SNNPR SCOPSO schools Denebe, Wanjana, Adilo Edola, Jore, Mirab Abaya, Molle, Kulfo, and Gero Non-SCOPSO schools Dalbo primary 88 U.S. Agency for International Development/Ethiopia Entoto Road, U.S. Embassy P.O. Box 1014, Addis Ababa, Ethiopia 89