STEPH PLEASE INSERT COVER PHOTO ABOVE THE BOX BELOW December 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by GRM International, on behalf of AMEX International Inc. EVALUATION EVALUATION OF THE OPPORTUNITIES FOR VULNERABLE CHILDREN PROGRAM INDONESIA iii | Page EVALUATION EVALUATION OF THE OPPORTUNITIES FOR VULNERABLE CHILDREN PROGRAM INDONESIA iv | Pag e This report was prepared by GRM International, on behalf of Amex International, contracted under: USAID Contract / Number: RAN-I-00-09-00008-00 Task Order / Number AID-497-TO-13-00003 Evaluation of USAID/Indonesia Program Opportunities for Vulnerable Children (OVC) Evaluation Team Members: Dr. Valerie Haugen, Budi Hermawan, Erlyn Yuli Astuti and Dawne Walker GRM Project Director: Oliver Mathieson Activity Start Date: August 12, 2013 Completion Date: December 31, 2013 Disclaimer The views expressed in this report are those of the authors and do not necessarily reflect the views of USAID, the Government of Indonesia, or any other organization or person associated with this program. v | Page CONTENTS ACKNOWLEDGEMENTS....................................................................................................................VII ABBREVIATIONS ...............................................................................................................................VIII EXECUTIVE SUMMARY .......................................................................................................................IX BACKGROUND...................................................................................................................................... IX THE OVC PROGRAM ............................................................................................................................. X OVC PROGRAM ACHIEVEMENTS........................................................................................................... XI LESSONS LEARNED AND RECOMMENDATIONS ......................................................................................XIII PART 1: INTRODUCTION...................................................................................................................... 1 BACKGROUND....................................................................................................................................... 1 THE USAID OVC PROGRAM ................................................................................................................. 3 PURPOSE OF THE FINAL EVALUATION..................................................................................................... 4 EVALUATION METHODOLOGY AND APPROACH........................................................................................ 5 EVALUATION LIMITATIONS ..................................................................................................................... 6 ORGANIZATION OF THE REPORT............................................................................................................. 7 PART 2: EVALUATION FINDINGS AND CONCLUSIONS................................................................... 8 OVERARCHING CONCLUSIONS ............................................................................................................... 8 HKI OVC PERFORMANCE AGAINST TARGETS.......................................................................................10 GOAL 1 ...............................................................................................................................................12 GOAL 2 ...............................................................................................................................................19 GOAL 3 ...............................................................................................................................................23 GOAL 4 ...............................................................................................................................................25 DISABILITY EQUALITY, SENSITIVITY TO GENDER, SEXUALITY, AND GENDER-BASED VIOLENCE...............30 SUSTAINABILITY ..................................................................................................................................35 PART 3: RECOMMENDATIONS .........................................................................................................37 LESSONS LEARNED .............................................................................................................................37 RECOMMENDATIONS............................................................................................................................38 ANNEX 1: STANDARD DOCUMENTATION.......................................................................................42 ANNEX 1.1: SCOPE OF WORK..........................................................................................................42 ANNEX 1.2: WORKPLAN ...................................................................................................................52 ANNEX 1.3: CONSULTATIONS...............................................................................................................53 ANNEX 2: METHODOLOGY................................................................................................................62 TYPE OF EVALUATION.......................................................................................................................62 vi | Page EVALUATION CONCEPTUAL FRAMEWORK .......................................................................................62 SAMPLING FRAME.............................................................................................................................62 METHODS ..........................................................................................................................................64 INSTRUMENTS ...................................................................................................................................66 PREPARATION FOR FIELDWORK, CONDUCTING FIELDWORK, AND PROCESSING OF DATA............67 RIGOR................................................................................................................................................68 CONSTRAINTS ...................................................................................................................................68 ANNEX 3: INSTRUMENTS ..................................................................................................................70 ANNEX 3.1: FGD INSTRUMENTS ......................................................................................................70 ANNEX 3.2: KEY PARTICIPANT INTERVIEW PROTOCOL................................................................. 101 ANNEX 3.3: SCHOOL ENVIRONMENT CHECKLIST ..........................................................................103 ANNEX 3.4: POLICY ANALYSIS INSTRUMENTS...............................................................................110 ANNEX 4: FINDINGS/CONCLUSIONS/RECOMMENDATIONS ANALYTICAL GRID ....................117 ANNEX 5: OVC PROGRAM PHASES...............................................................................................122 FIGURES FIGURE 1: N-OVC TARGETS VERSUS ACTUAL NUMBERS ......................................................................................12 FIGURE 2: POLICY CIRCLE FRAMEWORK...............................................................................................................13 FIGURE 3: NUMBER OF PROVINCIAL-LEVEL INCLUSIVE EDUCATION POLICIES ....................................................18 FIGURE 4: NUMBER OF DISTRICT-LEVEL INCLUSIVE EDUCATION POLICIES BY POLICY FOCUS AREA...................18 FIGURE 5: INCLUSIVE EDUCATION FRIENDLINESS BY SCHOOL TYPE ....................................................................28 FIGURE 6: NUMBERS OF BOYS AND GIRLS WITH SPECIFIC DISABILITY LABELS ENROLLED IN SCHOOL................31 FIGURE 7: ENROLLMENT OF CHILDREN WITH DISABILITIES BY SEX AND SCHOOL TYPE ......................................33 vii | Page ACKNOWLEDGEMENTS This report is the outcome of an evaluation study conducted by a team from GRM International on behalf of the United States Agency for International Development (USAID) and the Government of Indonesia. The research team is immensely indebted to various government and nongovernmental institutions that have provided both data and information and assistance with logistics and sourcing evaluation participants during the course of the evaluation. In particular, we are thankful for the contributions received from the following organizations and individuals: Organizations • Ministry of Education and Culture at the national, provincial, district, and school levels. Our particular thanks to Dr. Mudjito AK MSi, the Director of Special Education and Education Service Development, Basic Education Directorate General, Ministry of Education and Culture • National Development Planning Agency at the provincial level • Government of Aceh and governments of Banda Aceh, Aceh Besar, Lhokseumawe, and Pidie Districts • Government of East Java Province and the governments of Nganjuk and Tuban Districts • Government of DKI Jakarta and the governments of East Jakarta and South Jakarta Municipalities • Institut Pemerintahan Dalam Negeri (Institute of Public Administration) • Pertuni (Indonesian Blind Union) Individuals We would like to extend our appreciation to the parents of children with disabilities enrolled in the 38 primary schools visited, education personnel from those 38 schools, and district education personnel who gave their time for interviews and helped identify participants. In addition, we would like to thank USAID for the excellent collaboration with the evaluation team and its ongoing input and guidance as the evaluation was undertaken. viii | Page ABBREVIATIONS AIDS acquired immune deficiency syndrome DKI Jakarta Special Capital Region Jakarta FGD focus group discussion HIV human immunodeficiency virus HKI Helen Keller International ID intellectual disability IDR Indonesia rupiah IE inclusive education GOI Government of Indonesia LD learning disability MD multiple disabilities MOEC Ministry of Education and Culture MORA Ministry of Religious Affairs NA not applicable NB note well N-OVC New Opportunities for Vulnerable Children OVC Opportunities for Vulnerable Children (Program) PEPFAR President’s Emergency Response for AIDS Relief PRIORITAS Prioritizing Reform Innovation and Opportunities for Reaching Indonesia's Teachers, Administrators, and Students UNESA State University of Surabaya (Universitas Negeri Surabaya) UNICEF United Nations International Children’s Education Fund UNJ State University of Jakarta (Universitas Negeri Jakarta) USAID United States Agency for International Development USD U.S. dollars VI vision impairment ix | Page EXECUTIVE SUMMARY The Opportunities for Vulnerable Children (OVC) program represents a unique and highly commendable effort made by the United States Agency for International Development (USAID) to bring children with disabilities into Indonesia’s mainstream education settings in a substantive manner. The program—which only cost USD 9.3 million in 10 years or USD 930,000 a year on average—contributed significantly to strengthening the Indonesian enabling educational environment for children with disabilities, in the following ways: • Building substantial government bureaucratic and political support for inclusion of children with disabilities. • Spearheading the development of national policies, such as recognition of “resource teachers” as a legitimate category of service, and 22 subnational policies that operationalized the national inclusive education policies; these policies will continue to provide the legal framework for advancing system change, even though government leaders may change. • Helping stakeholders to lobby for and legitimize the development of budgets to support inclusive education, which will allow inclusive education initiatives to be funded post-OVC. • Changing parents’ and community attitudes about the rights of children with disabilities to an education. The above was achieved by well-targeted use of short-term technical experts and ongoing engagement of very knowledgeable program personnel who understand how to work with and within the existing systems. However, without strategic and selective external technical support for deepening and broadening OVC accomplishments, full realization of OVC program benefits is at risk. There is also a risk that inclusive education will remain synonymous with special education, rather than special education falling under the umbrella of a functional inclusive education system. USAID’s Education Strategy Goal 1 will ensure that 150 million children worldwide become readers in the early grades; however, most early grade reading assessments have not taken children with disabilities into account in testing protocols, data disaggregation, or ensuing programming. If USAID is able to act on the opportunity for disability-sensitive programming that the OVC program introduced, Indonesia can provide a valuable model on which to build in order to bring children with disabilities into this broader group of 150 million children. Failure of the government and donor community to further to deepen and extend OVC program achievements would be a significant and regrettable “missed opportunity.” BACKGROUND Of the 49 million Indonesian children between the ages of 7 and 18, around 13% or nearly 6.5 million children are not in school.1 These out-of-school children include those with disabilities, those who live on the streets, those who are migrants, and those who belong to ethnic minorities. In general, these children are affected by multiple dimensions of deprivation associated with poverty.2 The aggregation of data at the national level masks significant disparities in the education sector among and within 1 Ministry of National Development Planning (Badan Perencanaan Pembangunan Nasional or BAPPENAS), UNICEF, and SMERU Research Institute (2010), Child Poverty and Disparities in Indonesia: Challenges for Inclusive Growth (Jakarta: Indonesia: BAPPENAS). 2 Ibid. The six dimensions of deprivation include education, labor participation, health, shelter, sanitation, and water. Only about 18.3% of Indonesian children are free from all six deprivation dimensions. x | Page regions, provinces, districts, and schools in terms of gender, disability, and other vulnerabilities. 3 Opportunities for these children to access an appropriate education are extremely limited. Formal and nonformal education systems and secular and religious education institutions are, by and large, ill equipped to address the needs of all children, let alone children with very specific learning characteristics. Although the government is making efforts to transform traditional education into an inclusive education system, inclusive education continues to be equated with special education—a situation that presents both opportunities and problems. About 5% of the school-age population—or nearly 2.5 million children—have some type of disability.4 Children with disabilities who are unable to access a quality education and the absence of linkages and/or engagement with the world of work mean that these children grow into youth and adults unable to contribute to the economy. Donor interest in and funding for special education for children with disabilities—and children with special needs more broadly—has been limited in past decades. “Disability” as a cross-cutting issue in education projects has primarily been limited to some efforts to ensure that school construction is disability friendly, but even these initiatives have been limited in impact. Projects that specifically target special education for children with disabilities are few and far between, typically have extremely limited budgets and timeframes (sometimes as short as a year), and not surprisingly, have limited systemic impact. THE OVC PROGRAM In 2003 USAID responded to a request for funding from Helen Keller International (HKI) to implement the Opportunities for Vulnerable Children (OVC) Program, which ran from 2003 to 2006. The OVC program was intended to strengthen the inclusive education system in Indonesia, focusing specifically on enhancing inclusion of children with disabilities in the formal education system. USAID subsequently funded two additional unsolicited HKI proposals. The three funded phases of the program enabled HKI to continue to support the Government of Indonesia (GOI) in advancing inclusive education until February 2013 with an overall expenditure of USD 9.3 million. The OVC program used a multipronged framework consisting of the following goals: • Goal 1: Improve the coordination of policy, planning, and funding among the national, provincial, and district levels • Goal 2: Improve the capacity of universities • Goal 3: Improve in-service training programs • Goal 4: Increase awareness of inclusive education within the education system and the public HKI worked intensively with the Ministry of Education and Culture (MOEC) at national and subnational levels and with different units within the ministry, although special education units were the primary points of contact. HKI also engaged periodically with the Ministry of Religious Affairs (MORA). The OVC program’s phase 1 was intended to identify and address vision impairment as well as strengthen understanding at the local level and within the education sector on educating children with certain disabilities. OVC was initially implemented in five of Jakarta’s municipalities. OVC phase 2 and OVC phase 3 (the latter renamed New Opportunities for Vulnerable Children) expanded the efforts of 3 Ibid. and Ministry of Education and Culture (2013), Overview of the Education Sector in Indonesia 2012 Achievements and Challenges (Jakarta, Indonesia). 4 Per information obtained in 2010 from the Directorate of Special School Management, Directorate General Primary and Secondary Education, Ministry of National Education. xi | Page phase 1 by targeting additional areas of disability (hearing impairment, learning disabilities, and intellectual disabilities), increasing the scale of engagement with provincial and some district governments as well as schools in 35 districts in five additional provinces—Central Java/Yogyakarta, Aceh, South Sulawesi, East Java, and West Java. Phases 2 and 3 targeted initiatives to (1) increase enrollment of children with disabilities; (2) improve awareness at the family, community, and government levels about disabilities; (3) enhance teacher capacity; (4) accelerate policy development for inclusive education; and (5) strengthen capacity of six university special education departments to prepare student teachers for teaching children with learning disabilities, intellectual disabilities, and multiple disabilities. In mid-2013 USAID Indonesia contracted GRM International through AmEx International to carry out a final evaluation of the OVC program to examine program achievements and identify lessons learned to help inform current and upcoming education sector programming. A team of international and national education specialists carried out the evaluation study—completed in October 2013—using mixed qualitative and quantitative methods. In total, the team interacted with 308 stakeholders (185 females and 123 males)5 and visited 38 schools in 3 provinces and 6 districts. OVC PROGRAM ACHIEVEMENTS The OVC program has been effective and efficient in supporting the emerging transformation of Indonesia’s education system into an inclusive education system.6 The scope and scale of the three phases of the program were ambitious, particularly given the very limited budget, wide range of sites, and engagement across all levels of government with both bureaucrats and politicians and within and outside the education sector. Despite these challenges, few weaknesses were found in program implementation and accomplishments. Many program activities have been embedded within the government system. The following program approaches appear to have been important to successful implementation and resulting positive system change: • Building strong relationships with a wide range of stakeholders, from parents and community members to national and subnational levels of MOEC and MORA and development partners • Enabling individuals within the system to be leaders rather than followers • Providing structured opportunities for individuals to come together to learn, plan, and implement activities • Promoting a clear vision of an inclusive education environment and what it includes, and approaching barriers to such an environment from a range of angles • Implementing a clear vision on how to work effectively within the Indonesian context by: 5 The team made serious attempts to ensure that people with disabilities comprised part of the stakeholder pool but only managed to involve one female and one male individual with a disability. The difficulty in finding adults with disabilities in government positions likely reflects the lack of participation of people with disabilities in society more broadly. The one male parent with a disability was so ashamed that he would not participate in the focus group discussion to which he had been invited; he did agree to be interviewed separately. 6 It was not possible, for various reasons, to undertake a cost analysis; however, efficiency at the macro level strongly indicates that the OVC program accomplished a great deal on a small budget. xii | Page  Identifying opportunities and building on strengths (rather than focusing on weaknesses and attempting to address every gap)  Working within the system by using existing structures and processes  Showcasing success through effective advertising and positive competition The OVC program’s multipronged awareness-raising approach targeting government at all levels (including schools) and the general public (including parents) was especially effective. Raising awareness on what inclusive education is, the range of types of disabilities, and what rights children with disabilities have can make a tremendous difference in fostering system change. It is a testament to the success of the OVC program and the government in changing attitudes toward children with disabilities at all levels of the system that only one individual of 308 people interviewed made an overtly negative remark about mainstreaming children with disabilities in public schools. Most important, the OVC program has helped the government demonstrate that children with disabilities can participate successfully in mainstream public schools. Teachers reported that the training they received on how to work with children with disabilities enabled them to be more patient and that some of these children started to acquire reading and other skills. The benefits of inclusion have also extended to children without disabilities as teachers have developed a greater sensitivity to the learners they are teaching. Overall, some 65,000 children with and without disabilities benefitted from exposure to the 3,141 mainstream classroom and special education teachers who participated in some or all five phases of the inclusive education and classroom management professional development training. In DKI Jakarta alone, 4,700 children had their vision screened; 147 of these children had corrective eye surgery and 600 of these children received eyeglasses. 7 “With OVC, it was the right people in the right place at the right time!” said an East Java Department of Education administrator. The OVC program did miss some opportunities to strengthen the inclusive education sector in Indonesia. These include the absence of cost analyses for the program overall and particularly for the in-service teacher training model, and the construct for “identification, assessment, and enrollment” of children with disabilities. Although the government has institutionalized these two activities, the lack of available funding for all five phases of teacher training and for identification and assessment activities has limited capacity to implement them as planned, likely affecting the quality of scale-up efforts.8 An organizational development approach to strengthening university special education departments could have extended the benefits of the university capacity-building initiative beyond individual lecturers and mitigated some of the risks that resulted, including the lack of an ongoing system￾strengthening plan within the tertiary subsector. The policy development initiative enhanced the capacity of individuals, but the process lacked a review and revision focus; this gap increases the risk of ineffective or harmful policies. 7 Ministry of Education and Culture (2007), Perangkat untuk Mengembangkan Lingkungan Inklusif, Ramah terhadap Pembelajaran: Adaptasi Versi Indonesia, part of the UNESCO (2004), Embracing Diversity: Toolkit for Creating Inclusive, Learning-Friendly Environments (Bangkok, Thailand), which was translated by OVC, MOEC, and other development partners such as the Institute of Public Administration (Institut Pemerintahan Dalam Negeri). 8 Assessment should include access to (1) knowledgeable professionals with the expertise necessary to address disabilities that are more challenging to diagnose (such as learning disabilities and variation along the autism spectrum); (2) any medical intervention required to address physical issues (such as eye surgery) or mental health needs; and (3) appropriate assistive devices for learning and mobility that are updated annually as the child grows. xiii | Page LESSONS LEARNED AND RECOMMENDATIONS LESSONS LEARNED Lesson 1: Government leadership and commitment and a deep knowledge of the field and the local context can accelerate system change. Lesson 2: Positive competition and opportunities to gain recognition have the potential to raise the profile of inclusive education even more in the future. Lesson 3: Immediate attention to key risks is critical to continued success in increasing ac￾cess and quality to inclusive education. Lesson 4: Creative and committed action to improve coordination between MOEC and MORA is necessary to ensure children with disabilities and other special needs succeed. Lesson 5: Frequent transfers of competent government personnel threaten the sustainabil￾ity of government and donor investments and require immediate mitigating strategies. Lesson 6: A more holistic approach to strengthen capacity of special education departments would lead to greater impact and sustainability. Lesson 7: Lack of data and data integrity significantly inhibit sound policy development, planning, program implementation, student achievement, and development of an inclusive education system. Lesson 8: Filling the gap in high-quality and relevant research on inclusive education would foster evidence-based policy development, review, and revision at all levels of government. Lesson 9: Serious action is needed to address the higher levels of abuse among children with disabilities that negatively affects children’s education and later lives. Lesson 10: A donor champion such as USAID and a high-quality technical collaborator such as HKI through the OVC program are important allies in inclusive education system strengthening. RECOMMENDATIONS To deepen and extend the OVC program achievements, the following recommendations are offered. FOR THE GOVERNMENT OF INDONESIA Recommendation 1: Expand the OVC initiatives to ensure that the processes and products introduced in the past 10 years are taken further in terms of quality and quantity. Recommendation 2: Broaden the reach of inclusive education as the dominant education system, rather than “bolting on” an initiative for very vulnerable children that supplements the existing system. xiv | Page Recommendation 3: Undertake cost analyses of all interventions so that the different levels of government can sensibly expand the capacity to implement inclusive education based on a clear understanding of likely costs, depending on the location. Recommendation 4: Identify and put serious effort behind a way and means to enhance MOEC/MORA collaboration on inclusive education broadly and specifically children with disabilities. Recommendation 5: Develop and implement a capacity-building strategy and operational plan to strengthen the inclusive education task forces. Recommendation 6: Invest in strengthening research on inclusive education, from macro-level studies of the quality of policies and their effects to micro-level ethnographic research. FOR USAID Recommendation 7: Contract an OVC-type follow-on program with a systems-strengthening approach that builds on the OVC implementation model and incorporates a specific focus on the acquisition of reading skills by children with disabilities and opportunities for other vulnerable children. Recommendation 8: Appraise Prioritizing Reform Innovation and Opportunities for Reaching Indonesia's Teachers, Administrators, and Students (PRIORITAS) and other donor education project designs and approaches systematically and in-depth using an inclusive education lens and identify opportunities to anchor PRIORITAS and other donor education projects in the theory and practice of inclusive education. Recommendation 9: Draw on the extensive and high-quality work undertaken by the orphans and vulnerable children initiative under the USAID President’s Emergency Response for AIDS Relief (PEPFAR) to explore a more holistic approach to addressing the needs of very vulnerable children. Recommendation 10: Produce, publish, and present a case study on the Government of Indonesia and USAID OVC program experience and publicize program successes and lessons learned. 1 | Page PART 1: INTRODUCTION BACKGROUND Of Indonesia’s nearly 240 million people, 81.3 million are children 0 to 18 years of age (39.5 million girls and 41.9 million boys). Some 49 million of these children are between the school-age years of 7 and 18. Indonesia’s education system—the third largest in Asia and the fourth largest in the world— provides for more than 54.8 million students. The government has aspirations for achieving Education for All and Millennium Development Goals 2 and 3 (universal primary education and gender parity) by 2015. Currently, 83% of school-aged children are enrolled in primary or secondary school, and gender parity in primary education and junior secondary education is nearly at parity (although some disparities continue to exist in rural and remote areas).9 As Indonesia moves toward Education for All and Millennium Development Goals 2 and 3, the government is paying more attention to bringing children who are educationally disadvantaged and excluded (particularly poor children) into the classroom. Examples of equity strategies include national pro-poor programs such as the School Operational Assistance Program for primary and secondary schools, Operational Assistance for State Universities Program, and Scholarships for Poor Students. However, these programs do not necessarily or specifically target learners with disabilities in any substantive manner. In addition, the overlap between inclusive health and inclusive education (IE) is extremely limited and preventable physical conditions (such as blindness due to Vitamin A deficiency or deafness due to ear infections) are not taken into account in education programming; ultimately, this negatively impacts the child and his/her learning.10 The education system is at the very early stages of an anticipated transition from the dominant traditional educational paradigm in which children with disabilities are segregated from children without disabilities and educated in separate “special education schools.” At this point, efforts are very uneven across the country regarding teaching and learning processes and support materials, school infrastructure, and available and trained human resources—and consist of “bolting on” special education onto the traditional system in the form of designated inclusive education schools. This coupling of inclusive education with special education results in the perception that these two constructs are synonymous and thus causes issues that affect system transformation under an inclusive education paradigm and the specific considerations relevant to educating children with disabilities under that paradigm. Despite these successes, around 13% or nearly 6.5 million children are not in school and the aggregation of data at the national level masks significant disparities in the education sector across and within regions, provinces, districts, and schools regarding gender, disability, and other vulnerabilities. 11 These out-of-school children include, among others, those with disabilities, those 9 Ministry of National Development Planning (BAPPENAS), UNICEF, and SMERU Research Institute (2010), Child Poverty and Disparities in Indonesia: Challenges for Inclusive Growth (Jakarta: Indonesia: BAPPENAS). 10 According to UNICEF’s 2013: Children with Disabilities (New York, NY), the “paucity of data on children with disabilities has contributed to a misconception that disability does not merit global priority. General data collection instruments—such as census or household surveys—are likely to underestimate the number of children with disabilities.” The actual number of Indonesian children with disabilities disaggregated by sex and disability type is not known. 11 UNICEF (2013), State of the World’s Children 2013: Children with Disabilities (New York, NY); Ministry of Education and Culture (2013), Overview of the Education Sector in Indonesia 2012 Achievements and Challenges (Jakarta, Indonesia). 2 | Page who live on the streets, those who are migrants, and those who belong to ethnic minorities. Half of Indonesian children live in families in the bottom two income quintiles. In general, these children are affected by multiple dimensions of deprivation12 associated with child poverty and fewer than 1 in 5 or about 18.3% of Indonesian children are free from all of the 6 deprivation dimensions. In addition, children are not evenly distributed across Indonesia: 54% live in Java where poverty rates are nearly 50% and 54% live in rural areas where the poverty rate is higher than for children in urban areas.13 To ensure that all children develop to the best of their ability, the government is now using a more holistic approach that expands the scope of child protection into all aspects of national development, including education and basic health.14 The cited number of children with disabilities varies; however, the commonly accepted figure of 1.5 million children between the ages of 0 and 18, or a 2% share of all children, is derived from the 2010 Census. 15 Given the high correlation between poverty and disability, these children are found in greater numbers in poor families, which have the least capacity to accommodate the needs of their children.16 And, despite growing government support for special education for children with disabilities (and special needs education more broadly), the education sector struggles to provide opportunities for children with disabilities to access an appropriate education during their school years. The formal and nonformal education systems and secular and religious education institutions are, by and large, ill equipped to address the needs of all children, let alone children who have very specific learning characteristics. The government has worked to increase the availability of qualified teachers, especially in remote and rural areas, in order to reduce regional disparities in educational attainment as well as levels of welfare and incomes. At present, the government employs some 3 million teachers who work in 236,000 schools across the country17 and relies greatly on the 40% of teachers who are contract teachers (guru honor), whose salaries are substantially lower than those of teachers who are civil servants. Indonesia has one of the lowest student-teacher ratios in the world, and since 2004, the number of teachers has continued to outgrow the number of students; however, few teachers in mainstream classrooms are equipped to teach children with various types of disabilities that may affect the child’s academic success. 18 Teachers who are starting to be used as itinerant resource teachers for inclusive education schools are predominantly contract teachers and may not be able to address the full range of learning needs of children with disabilities, even if they have received 12 Ministry of National Development Planning (BAPPENAS), UNICEF, and SMERU Research Institute (2010), Child Poverty and Disparities in Indonesia: Challenges for Inclusive Growth (Jakarta: Indonesia: BAPPENAS). 13 Ibid. The figure for rural areas is slightly larger than the 52% of the overall population living in rural areas (2009 data). 14 Child protection issues are incorporated into the 2003 National Education Law, 2009 Health Law, and 2003 Labour Law, as well as through a more integrated approach to child protection adopted in the current National Medium-Term Development Plan (RPJMN 2010–2014). 15 Government of Indonesia (2010), National Household Census (Jakarta, Indonesia). Given that the World Health Organization and World Bank (2011) World Report on Disabilities notes that 15% of a population typically has some type of disability and given challenges in identifying and counting individuals with disabilities in developing countries, the 1.5 million figure is very likely to be much higher. 16 UNICEF (2013), State of the World’s Children 2013: Children with Disabilities (New York, NY). 17 Ministry of Education and Culture (2013), Overview of the Education Sector in Indonesia 2012: Achievements and Challenges (Jakarta, Indonesia). 18 World Bank (2013), Spending More or Spending Better: Improving Education Financing in Indonesia, Policy Brief 76404 (Washington, DC and Jakarta, Indonesia). There is a trend toward the “regularization” of contract teachers to civil servant status. Converting all teachers to civil servants would increase the teacher salary bill by at least 50%, and providing certification allowances would more than double the salary bill. The World Bank notes that such policy changes would not be financially feasible. 3 | Page additional training. New teacher education graduates are also unprepared for teaching the full range of children with disabilities who are increasingly entering the traditional education system. The government’s “20% rule” is based on a 2002 amendment to the Constitution19 mandating at least 20% of the total government budget to education. Between 2006 and 2011, the education share of national expenditures grew by 180% from 7.2% to 20.2%; the education share of the gross domestic product increased between 2006 and 2010 by 142% from 1.4% to 3.4%. Nonetheless, funding for special education and special services is constrained at all levels, despite increases in allocations to these areas. Donor interest in and funding for special education for children with disabilities—and children with special needs more broadly—have been limited in past decades. “Disability” as a cross-cutting issue in education projects has primarily been limited to some efforts to ensure that school construction is disability friendly, but even these initiatives have been limited in impact. Projects that specifically target special education for children with disabilities are few and far between, typically have extremely limited budgets and timeframes (sometimes as short as a year), and not surprisingly, have limited systemic impact. Children with disabilities who are unable to access a quality education and lack linkages and engagement with the world of work mean that they grow into youth and adults who are unable to contribute to the economy. THE USAID OVC PROGRAM In this context, the United States Agency for International Development (USAID) responded to an unsolicited request for funding from Helen Keller International (HKI) to implement the Opportunities for Vulnerable Children (OVC) Program from 2003 to 2006. The OVC program was intended to strengthen the inclusive education system in Indonesia, focusing specifically on enhancing inclusion of children with disabilities in the formal education system. USAID subsequently funded two additional unsolicited HKI proposals. The three funded phases of the program enabled HKI to continue to support the Government of Indonesia in advancing inclusive education until February 2013 with an overall budget of USD 9.3 million. The OVC program collaborated intensively with the Ministry of Education and Culture (MOEC) at the national level, particularly the Special Education and Special Services Directorate (Pendidikan Khusus dan Layanan Khusus) and subnational levels. OVC engaged periodically with the Ministry of Religious Affairs (MORA). The program used a multipronged framework consisting of the following goals: • Goal 1: Improve the coordination of policy, planning, and funding among the national, provincial, and district levels • Goal 2: Improve the capacity of universities • Goal 3: Improve in-service training programs • Goal 4: Increase awareness of inclusive education within the education system and the public OVC program phase 1 targeted vision impairment as well as strengthened understanding at the local level and within the education sector on educating children with certain disabilities. The program initially implemented this phase in five of Jakarta’s municipalities. OVC phase 2 and phase 3, the 19 Government of Indonesia (2002), Amendment to the 1945 Constitution (Jakarta, Indonesia). 4 | Page latter renamed New Opportunities for Vulnerable Children (N-OVC), expanded the efforts of phase 1 by targeting additional disability areas (hearing impairment, learning disabilities, and intellectual disabilities); It also successively increased the scale to engage with provincial and some district governments as well as schools in 35 of Indonesia’s 455 districts in five additional provinces: Central Java/Yogyakarta, Aceh, South Sulawesi, East Java, and West Java. Phases 2 and 3 targeted initiatives to (1) increase enrollment of children with disabilities; (2) improve awareness at the family, community, and government levels on disabilities; (3) enhance teacher capacity; (4) accelerate policy development for inclusive education; and (5) strengthen capacity of six university special education departments to prepare student teachers for teaching children with learning disabilities, intellectual disabilities, and multiple disabilities. PURPOSE OF THE FINAL EVALUATION As part of USAID Forward, the agency produced the USAID Education Strategy 2011–2015. The strategy revolves around three overarching goals: • Goal one: Improved reading skills for 100 million children in primary grades by 2015 • Goal two: Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country’s development goals • Goal three: Increased equitable access to education in crisis and conflict environments for 15 million learners by 2015 USAID missions are expected to align education programming under one or more of these goals. USAID Indonesia programming is aligned under goals 1 and 2; consequently, the Mission will be contributing to reaching the targeted 100 million new readers and improving the quality of and access to post‐primary education, particularly for learners in vocational schools polytechnics and community colleges. Education programming falls under the USAID Indonesia Country Development and Cooperation Strategy (2013–2018) Development Objective 2, “Essential Services for the Poorest and Most Vulnerable Improved.” The Prioritizing Reform for Indonesia’s Teachers, Administrators and Students (PRIORITAS) program, which began in 2012 and will finish in 2017, is being adapted to partially align with goal 1. The Mission is interested in helping to ensure that the Government of Indonesia continues to strengthen inclusive education and is particularly interested in the potential to incorporate aspects of OVC programming into PRIORITAS. Consequently, in mid-2013, USAID Indonesia contracted GRM International through AmEx International to carry out a final evaluation of the OVC program to examine program achievements and identify lessons learned to help inform current and upcoming education sector programming. Enhancing USAID’s and the government’s understanding how the OVC program fostered change in the education system and supported the shift toward inclusive education was an important dimension of the evaluation. USAID/Indonesia requested that the evaluation team examine four broad evaluation questions: • What aspects of the OVC program proved most and least effective in creating a more inclusive environment for each of the OVC program goals? 5 | Page • How inclusive is the environment for children with special needs among program communities? • What evidence exists that the program activities and results might be sustainable after completion of the program itself? • To what extent did the OVC program account for and address the gender-specific challenges and needs of girls and boys? EVALUATION METHODOLOGY AND APPROACH The final evaluation study of OVC was carried out in October 2013 by a team of international and national education specialists: a team leader with education evaluation experience, a local inclusive education specialist, a local subcontractor for data collection and analysis, and additional intermittent support from a monitoring and evaluation specialist. USAID’s evaluation questions guided selection of the evaluation methods and participants. The team used mixed qualitative and quantitative methods and met with the USAID management team to discuss the study design. Site visits were made to 3 of the 6 OVC provinces and 2 districts in each province. One province for each of the OVC phases was selected. In the case of phase 1, OVC only worked in DKI Jakarta. Aceh was selected over South Sulawesi because of the province’s unique characteristics and East Java over West Java on the basis of ease of access for the local research team. Districts were also selected purposely based on the following criteria: the most successful and active district and the least successful and active district. The evaluation team relied on OVC documentation and the perspectives of government, USAID, and HKI personnel to guide the selection. Five schools in each of the six districts were also selected according to the following design in order to test out the center-to-periphery cluster (gugus) model used by the government and used and extended for OVC implementation. It entailed a special education school linked with (1) an OVC model (government inclusive-education main) school, (2) OVC satellite (government inclusive￾education satellite) schools (two), and (3) a government satellite school (not part of OVC and not a government-designated inclusive-education school). Government (district-level) personnel selected schools according to the evaluation team’s request to select the best example of the OVC cluster model. For informants at the community and school levels, district and school personnel received a set of criteria to ensure that the groups of parents and teachers were representative. Selection of informants at the district, provincial, and national government levels was straightforward and entailed looking for people in key positions. In total, the team visited 38 schools in 3 provinces and 6 districts and interviewed 308 individuals (including 186 females including 1 female with a disability and 123 males including 1 male with a disability) who ranged in age from 27 to 60 years. Respondents were drawn from the following groups: parents of children with disabilities, primary school personnel, district and provincial leaders, MOEC staff, donor and development partner personnel, and HKI staff. The team made serious attempts to ensure that people with disabilities made up part of the stakeholder pool but only managed to involve one female and one male individual with a disability. The one male parent with a 6 | Page disability was so ashamed that he would not participate in the focus group discussion to which he had been invited; he did agree to be interviewed separately.20 The majority of the primary data were qualitative and derived through focus group discussions (FGDs), key informant interviews, and direct observation. The evaluation team used a structured inteview guide to conduct the FGDs and a semistructured questionnare to guide the key informant interviews. Quantitative data were collected using a structured checklist to examine school infrastructure and instructional elements important for children with disabilities. The team used thematic coding to analyze the FGD and key informant interview data. The school environment checklist captured “yes/present” and “no/not present” responses to the set of school environment elements under consideration on a paper-based form. These responses were then entered into an Excel workbook that mirrored the checklist and the data were analyzed. Secondary source documents and data (processed and raw) from the OVC program including quarterly reports, final program reports, routine data, and internal evaluation reports were also reviewed. These documents and data supplemented the primary data collected and analyzed. EVALUATION LIMITATIONS Lack of baseline data or targets. Two major limitations of the evaluation were the lack of baseline data for the different phases of OVC and the lack of targets for program indicators. The evaluation questions state that the evaluation team should determine the program’s most and least effective activities; however, without reliable baseline data collected at the start of the OVC program, it was difficult to directly attribute impacts and/or identify contributions of program efforts. Time constraints. Time was an issue during each phase of the evaluation. The period from the initiation of the study to the completed report was about 14 weeks. This is a limited period in which to review 10 years of program documentation; develop, obtain feedback on, and revise a sound methodology and instruments for qualitative and quantitative data collection; train the evaluation team; test and revise instruments; collect, transcribe, clean, and analyze data; provide quality assurance; and draft, obtain feedback, revise, finalize, and translate into Bahasa Indonesia the following documents: the evaluation report plus annexes, a lessons learned report, a press release, and a PowerPoint presentation. Planning and managing the logistics of the evaluation study was also very time consuming. Program considerations. Gaining access to in-depth information about OVC implementation from OVC personnel was problematic. The program had ended some months earlier, and many key personnel (particularly field staff) were not available for interviews. Gaining access to program stakeholders was also difficult and very time consuming without being able to rely on program personnel for assistance. The monitoring and evaluation system, including documentation and program monitoring plans, was not an adequate substitute for in-depth face-to-face interviews. 20 The difficulty in finding adults with disabilities in government positions likely reflects the lack of participation of people with disabilities in society more broadly. 7 | Page ORGANIZATION OF THE REPORT This report is organized as following: Part 1: Introduction presents a short analysis of the inclusive education situation in Indonesia, particularly regarding children with disabilities. It also describes the OVC program and summarizes the evaluation methodology. Part 2: Evaluation Findings and Conclusions begins with a set of overarching, high-level conclusions informed by and based on content contained in subsequent sections of the report. This part also presents a synopsis of OVC program achievements based on a thorough review of project documentation and provides key findings and conclusions of the evaluation study in terms of the program’s four goals. The key findings and conclusions are informed by and based on the evaluation study primary data, secondary source documentation from the Government of Indonesia, USAID, international literature, and the collective experience of the evaluation team in the Indonesian context and internationally. Part 2 also discusses the specific areas of (1) disability equality, sensitivity to gender, sexuality, and gender-based violence and (2) sustainability of program interventions and benefits. Part 3: Recommendations begins with a set of lessons derived from reflection on the findings and conclusions of the evaluation study and presents a set of recommendations each for the Government of Indonesia and for USAID. A number of annexes are included for the reader’s reference as well. 8 | Page PART 2: EVALUATION FINDINGS AND CONCLUSIONS Part 2 begins by presenting a set of overarching conclusions and a comparison of HKI performance against targets; these are informed by and based on the findings presented in part 2 and lessons learned presented in part 3. The following four sections then describe key findings and conclusions of the evaluation study for each of the program’s four goals, based on respective sets of questions that USAID requested the evaluation team to address. These findings and conclusions are informed by and based on the evaluation study’s primary data; secondary source documentation from the Government of Indonesia, USAID, and international literature; and the collective experience of the evaluation team in the Indonesian context and internationally. Part 2 concludes with a discussion of disability equality, sensitivity to gender, sexuality, and gender-based violence, as well as the sustainability of the OVC program’s work. OVERARCHING CONCLUSIONS Given the scope, scale, and variety of interventions under the OVC program, the team concludes that HKI has done a highly commendable job with a limited budget and limited personnel in advancing the institutionalization of inclusive education, specifically inclusive education that targets children with disabilities where the program was implemented. OVC has been a valuable and well-appreciated partner in expanding elements that are fundamental to an inclusive education system. Some of the OVC initiatives will continue to have positive impacts as the government’s capacity and sophistication in mainstreaming inclusive education is increasingly the preferred educational paradigm. OVC multiplier effects emerged during the program and continue to be observed: • Noticeable expansion in the number of children with disabilities who are able to participate in mainstream schools and classrooms • Increase in parents who are better informed about the type of disability their child has, what their child needs, and what rights their child has • Emerging professionalization in teaching children with disabilities beyond expertise typically confined to specific education institutions for children with special needs • Strengthening of the policy framework, which will help ensure that changes in government do not affect inclusive education gains. In terms of the four OVC goals listed on page 3, it is clear that the program had significant success in addressing each goal. It is equally clear that some things could have been done differently without going over budget. However, it is not possible to conclude that one goal was more important than another: the inclusive education system consists of interdependent parts and to neglect one part has implications for the other parts. It is possible to conclude that sequencing of and levels of emphasis on given initiatives may be particularly important to take into consideration. The barriers to inclusive education are consistent across the archipelago; what is not consistent is the magnitude of the various barriers. Consequently, although a structured, sequenced model can work, the challenge is in adjusting the treatment of various initiatives within the model to fit the strengths and needs of a specific localized context (province, district, or school). An illustration of this concept is found in community awareness raising. This initiative was so successful in one program location that it had to be stopped because the local education system could not cope with the huge increase in children trying to access schooling. 9 | Page It is important to point out that an “inclusive environment” can mean many things and can be examined from a variety of angles, particularly regarding enhancing education for children with disabilities. Addressing the range of characteristics, strengths, and weaknesses that are found in children with disabilities is a complex challenge in and of itself. In 1990 the Directorate General for Education said that all children should have access to education, but according to HKI no significant progress was made on inclusive education until 2003; before this, more children with visual impairment were out of school than in, and most children with disabilities were in special schools. OVC initially focused their efforts on children with visual impairment in DKI Jakarta. The selection of the additional provinces—Aceh, Central Java, East Java, South Sulawesi, West Java, and Yogyakarta—in subsequent phases was based on the presence of the USAID-funded Decentralizing Basic Education Project among other factors. OVC was also colocated by default with the Strengthening Education in Aceh program (funded by the Australian Agency for International Development). The following attributes of the OVC approach have been important in creating a more inclusive environment: • Working across different government levels of the education sector and drawing in key personnel from other sectors as well as politicians • Providing structured opportunities for individuals to come together to learn, plan, and implement activities • Working with a wide range of stakeholders from parents and community members to the national level of the Ministry of Education and Culture and other development partners • Promoting a clear vision of an inclusive education environment and what it includes, and approaching barriers to such an environment from a range of angles • Implementing a clear vision on how to work effectively within the Indonesian context:  Identifying opportunities and building on strengths (rather than focusing on weaknesses and attempting to address every gap)  Working within the system by using existing structures and processes  Showcasing success through effective advertising and positive competition The benefits of OVC colocation with another education program are not obvious and the assumptions underpinning colocation do not appear to have been tested or reported on to any great extent. Participants in Aceh did mention that OVC collaborated with the Strengthening Education in Aceh program to bring inclusive education into provincial and district planning; however, they were not able to give any specific examples. OVC collaboration with Save the Children in Aceh enabled the introduction of OVC initiatives in four additional districts for a year. The absence of real or assumed benefits from OVC colocation and collaboration with other donor programs has implications for USAID’s five-year PRIORITAS project. If colocation and collaboration did not bring obvious benefits during OVC implementation, exploiting OVC work after the fact will likely be even more challenging. 10 | Page HKI OVC PERFORMANCE AGAINST TARGETS During the three phases of OVC, the program grew, not just in terms of budget allocation, but also in terms of the breadth of disabilities it covered and the number of provinces and districts that it supported. Under phase 1, the program focused on children with visual impairment (VI) in DKI Jakarta. It was an urban model whose main objective was to provide technical assistance to schools for the blind to strengthen their management capacity to provide better education services and to develop models of schools for the blind that were state of the art and culturally sensitive. In addition, OVC focused on creating public awareness of the rights of children with visual impairment. HKI-Identified Accomplishments of OVC: Phase 1 1 Promoted shifts in policy allowing for more blind and low-vision children to access schooling 2 Increased the number of official inclusive education schools by 69% 3 Increased enrollment of children with severe to low vision and blindness in education programs by 300% 4 Supported vision checks of 4,700 children in partner schools, through which 600 children received eyeglasses and 147 children with severe low vision had their vision restored through eye surgery 5 Collaborated with the government to develop the National and Presidential Focus for Inclusion 2005 6 Collaborated with DKI Jakarta to develop and progress an advanced public early-intervention program for the disabled 7 Collaborated with DKI Jakarta to develop a foundation for support services and to create mobile resource teachers and resource centers for improved services for visually impaired children Under phase 2, the OVC program expanded its geographic reach to Aceh, Central Java, and South Sulawesi, and also expanded use of its model from urban to peri-urban and rural areas. OVC also expanded its definition of beneficiaries to include children with hearing impairment and learning disabilities. In Jakarta, OVC continued to train resource teachers to work with children with disabilities, including those with visual impairment, intellectual disability (ID), and hearing impairment. In Aceh and Central Java, OVC collaborated with the Provincial Department of Education to develop provincial strategies for inclusive education. 11 | Page HKI-Identified Accomplishments of OVC: Phase 2 1 Trained 241 resource teachers 2 Trained 2,949 mainstream teachers and government counterparts in inclusive education 3 Supported the development and passage of 22 policies 4 Advocated the importance of inclusive education with government, leading to allocation by provincial and district governments of an estimated USD 137,167 to inclusive education 5 Established two satellite early intervention and resource centers in Jakarta 6 Launched the DKI Jakarta public awareness and participation “Towards Inclusion” campaign 7 Piloted the preservice university program at Makassar State University to strengthen the capabilities of new teachers to teach children with visual impairment N-OVC—phase 3 of the OVC program—ran from 2010 to 2013. N-OVC added East Java and West Java as supported provinces and expanded the focus area to target policy development at the national level while continuing to support policy development at the subnational level. The program also continued with preservice training for teacher education for special education, in-service teacher training, and advocacy for inclusive education. Key achievements under N-OVC included local government support for inclusive education beyond the districts directly supported by N-OVC, provincial and district budget allocation for inclusive education, and development and adoption by the Special Education and Special Services Directorate of the National Inclusive Education Awards. HKI Identified Accomplishments of N-OVC: Phase 3 1 Motivated the Special Education and Special Services Directorate (PPK-LK Dikdas) to replicate the N-OVC IE model in 10 districts in 10 provinces outside the N-OVC partner provinces 2 Supported development of 20 inclusive education model schools in 4 districts in East Java and West Java 3 Collaborated with local governments to develop and issue governors’ decrees for Aceh, East Java, and South Sulawesi on IE allowing provincial departments of education and BAPPEDA21 to sustain IE initiatives 21 Agency for Regional Development and Planning. 12 | Page 4 Trained 23 teachers as national mentor teachers, 13 of whom actively engaged in delivering training specific to learning disability (LD), intellectual disability (ID), VI, and emotional impairment (EI) 5 Trained 23 lecturers from 6 N-OVC partner universities on LD/ID and EI/MD/VI 6 Developed a new LD/ID and EI/MD/VI syllabus, which is used by the six partner universities 7 Adoption of the Inclusive Education Award by the Special Education and Special Services Directorate, MOEC 8 Conducted 13 radio talk shows in Aceh, East Java, Jakarta, South Sulawesi, West Java, and Yogyakarta, as well as TV talk shows which USAID personnel participated in For N-OVC, HKI established targets for their program indicators. N-OVC almost reached most of their targets. The program came up short on two indicators: the number of inclusive primary schools and the number of resource teachers trained (figure 1). Figure 1: N-OVC Targets versus Actual Numbers GOAL 1 Improve the Coordination of Policy, Planning, and Funding among the National, Provincial, and District Levels KEY FINDINGS The evaluation team used the Policy Circle Framework22 illustrated in figure 2 to analyze the OVC policy initiative in order to arrive at key findings. 22 The USAID Health Policy Project developed the Policy Circle Framework. 13 | Page Figure 2: Policy Circle Framework Figure 2 addresses the standard elements of policy development—problem, people, process, price tag, paper, and programs—and depicts the way these elements interact in a nonlinear manner. Each element involves the following: • Problem: problem identification • People: government, nongovernmental organizations, communities, and individuals • Process: policy development (issue framing, agenda setting, data analysis, advocacy, dialogue, and formation) • Price tag: the cost of policy options and resource allocation • Paper: actual laws and policies • Program: results from implementing policies and their performance in achieving progress toward goals and objectives This framework also takes into consideration that policies are produced in a specific enabling or disabling political, social, and economic environment. Problem identification. Indonesia has a decentralized government system that gives the subnational government autonomy in establishing local regulation and decrees. Both before and during implementation of the OVC program, policy makers encountered specific barriers that, although not unique to the education sector or inclusive education, nevertheless had to be addressed. Government personnel who participated in the evaluation consistently mentioned the following issues: Local governments must comply with laws and regulations passed by the national parliament, but they need not adhere to ministerial regulations and decrees if these contradict local policies. For example, even though the national government has enacted IE policies, this does not mean that provincial or district governments will adopt them through provincial or district decrees. (Although according to our analysis (see the subsection below entitled “Paper”), several national decrees were adopted at the local level.) At the subnational level, regulations and decrees passed by the provincial or district governments are more permanent than those passed by an individual governor, mayor, or district head. A decree passed by the governor, mayor, or a district leader can be overturned when a new leader comes into power; whereas the local parliament must vote to overturn legislation passed by the government. 14 | Page “As it is more difficult to enact a provincial or district regulation as they have to be passed by local parliaments, most IE regulations at local level were passed as governor/mayoral/bupati regulations. The issue with this is that when a new governor or mayor comes into power, these regulations can easily be changed.” —Deputy Director for Curriculum and Learning, Special Education and Special Services Directorate MOEC “HKI staff were able to facilitate coordination among stakeholders from education, planning (cooperation bureaus), and local parliaments and promote inclusion in our educational programs and establishment of the Governor Regulation and Provincial IE Grand Design. We still use former HKI staff to help us develop IE strategy." —Director, Special Education Section, Provincial Department of Education, East Java Policy makers often lack a comprehensive understanding of inclusive education and some government officials do not know how to develop inclusive education policies and cannot do so without technical assistance. “Even though there is Ministerial Regulation No. 70, 2009 about inclusive education, we did not have capacity to develop the regulation on IE, but OVC provided us with the assistance to do so” (head of Curriculum Section, Basic Education Division, Jakarta Provincial Department of Education). At times, some government entities at different levels of the system overlook or forget about inclusive education. "Actually the district department of education very often forgets what inclusion is. Your (evaluation team) coming here helped us to remember the concept. The transfer of education leaders due to political affiliation contributed to this issue as new leaders would come in and change the policy” (senior staff member from Lhokseumawe District Department of Education). Sociocultural attitudes, beliefs, and norms on disability are antithetical to inclusion. There is tremendous stigma associated with disabilities. In many parts of the country, parents are ashamed of having a child with a disability and hide the child from others. Some people believe that they are being punished for a sin they or another family member have committed if they have a child with a disability. People (policy stakeholders). Through support from the OVC program, the Special Education and Special Services Directorate and the Directorate General of Primary and Secondary Education, signed off on the national inclusive education program. The task force has two segments: an expert team comprising university lecturers and a technical team comprising MOEC staff. Development partners, including nongovernmental organizations and donor organizations, are only invited to meetings when there is a specific issue that relates to them or when they can provide specific expertise. All three of the provinces visited by the evaluation team had IE task forces that had been established through technical assistance provided by OVC. In DKI Jakarta, the government authority is located at the provincial not district level, so the IE task force only operates at the provincial level. Inclusive education task forces are operating in all provinces in which HKI implemented the OVC program. Task force members come from the provincial education authority, universities, Disabled People Organization, nongovernmental organizations, and planning offices. The head of the education 15 | Page authority at the provincial level has formalized the task forces, which are tasked with coordinating implementation of the inclusive education implementation plan. Three of the other four districts visited by the evaluation team also had IE task forces. The OVC program increased the capacity of IE task force members by including them in their training and workshops at the national and provincial levels. At the time of the evaluation, none of the task forces in the three provinces visited by the team had budgets allocated. This lack of financing was one of the main reasons given for not meeting on a regular basis. According to HKI, restructuring of MOEC led to delays in working with personnel on strengthening the inclusive education policy environment. MOEC personnel were uncertain about their roles and responsibilities regarding IE, so they were uncertain about which stakeholders OVC should work with on national-level policy development. There were also indications of limited collaboration with and commitment from MORA around OVC initiatives. Process (policy development). OVC advocacy and lobbying efforts with makers’ ideas on inclusive education, which led to the enactment of decrees (detailed below in the subsection titled “Paper”). In a focus group discussion, staff from the provincial department of education, BAPPEDA, and the National and International Cooperation Bureau, East Java, agreed with one member’s statement that “We need an INGO [international nongovernmental organization] like HKI because they can provide us with a resource person not ‘money’ for building capacity of our teachers and government officers to promote inclusion in our schools, districts, and province.” Similar comments were made by education stakeholders in Aceh and by the deputy director for curriculum and learning, Special Education and Special Services Directorate, MOEC). According to a deputy director within MOEC, OVC contributed to the policy environment for IE, especially at the provincial and district level. OVC’s advocacy efforts contributed to the development of the model school approach and helped build capacity to identify children with disabilities. In addition, with support from the national government, OVC through its advocacy work promoted a “grand design” with provincial and district governments. The head of the section for special education in East Java told the evaluation team that the IE program will move forward because they have developed provincial grand design documents based on the model presented by the OVC program. In East Java, OVC collaborated with the provincial government to develop the Governor Regulation on Inclusive Education, as well as the Head of Education Authority Decree to appoint several schools as inclusive schools. Before the OVC program, fewer than 400 inclusive schools existed in East Java, afterward, there were 755. East Java also designated the Provincial Feeder Special School in Malang as an IE resource center. Through their work with the Planning Department in East Java, OVC’s advocacy work was instrumental in supporting passage of a new provincial regulation on the rights of persons with disabilities, which would require their inclusion in all sectors of government. Through OVC’s advocacy efforts in Aceh, SLB A-B YPAC Banda Aceh special school was appointed as an IE resource center for Banda Aceh and Aceh Besar. Through their advocacy work with MORA in Aceh, the provincial government began to promote inclusion in the madrassahs. Lobbying efforts in Aceh have also led to the provincial government allocating money for IE in 2014. “There were not many development agencies as committed as HKI in promoting coordination among stakeholders in East Java. The HKI program was well known by most stakeholders in our province.” —BAPPEDA Functional Staff and Cooperation Unit senior staff 16 | Page “Our budget increased dramatically since we adopted OVC. We started the program called ‘Socializing Inclusive Education Movement’ that we use to provide national grants totaling USD 90,000 / IDR 900,000,000 for the 40 most committed districts and the four most committed provinces.” —Director, Special Education and Special Services Directorate, MOEC) The head of special and out-of-school education in the DKI Jakarta Department of Education stated that an increased number of schools were motivated by OVC to work toward inclusion. In 2011 there were 164 inclusive education schools and by 2013 there were 224 along with 21 special education schools designated as resource centers. In DKI Jakarta, OVC was able to work with the government to have all special schools designated as resource centers. Price tag. According to government evaluation participants from the national, provincial, and district levels, because of the OVC lobbying and advocacy work mentioned above, the government started to allocate new or increased budgets for inclusive education in phase 3. All three OVC provinces and five of the six districts visited by the evaluation team were committed to continuing their budget provision for IE and annually increasing the allocations. For example, the provinces of Aceh, DKI Jakarta, and East Java targeted have committed to providing budget allocations until 2015 under their inclusive education grand designs, which run from 2011 to 2015. The director of special and nonformal education in the Aceh Provincial Department of Education said, “OVC-HKI was the most active partner in drafting and promoting our governor or head of district regulations on inclusive education which will promote inclusive education budget allocations.” The director of basic education, East Java Provincial Education Department made a similar comment. At the national level, the Special Education and Special Services Directorate was allocated a national budget of IDR 19.5 billion (USD 1.95 million) for inclusive education in 2010. In 2011 the IE budget nearly tripled to just more than IDR 56 billion (USD 5.6 million). The budget decreased to IDR 29 billion (USD 2.9 million) in 2012. At the provincial level, DKI Jakarta allocated IDR 7 billion (USD 700,000) in 2012 for IE and IDR 9 billion (USD 900,000) n 2013. East Java had a flat budget allocation in 2012–13 allocating IDR 2.5 billion (USD 250,000) a year for IE. Lobbying efforts in Aceh have led the provincial government to allocate money for IE in 2014. Although amounts were small at the district level, district governments working with OVC had started to allocate budgets for IE. In Nganjuk District (East Java Province), the district government allocated IDR 35 million (USD 3,500) in 2012 for IE socialization and IDR 300 million (USD 30,000) in 2014. Tuban District, also in East Java Province, allocated IDR 75 million (USD 7,500) for implementation of IE at the district level. The district also funded a workshop for training 20 resource teachers in IE during 2013. Pidie District (Aceh Province) designated IDR 44 million (USD 4,400) for IE in its 2013 budget. Other OVC districts did not allocate a budget, for example, Lhokseumawe District. This finding mirrors that of the USAID policy environment situation analysis.23 The reasons given by evaluation participants include changes in leadership and lack of support among new leaders who 23 USAID (2013), Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students, (USAID PRIORITAS) Inclusive Education Policy Study (Jakarta, Indonesia). 17 | Page “HKI is one of the INGOs [international nongovernmental organizations] that successfully promoted the establishment of Governors regulations, Bupati’s Regulations and/or Mayor Regulations in the regions in which OVC worked. The Governor and District leader regulations were formalized into a standard template, which has been used to develop other IE provincial and district leader regulations in the 40 IE Districts awarded grants by MOEC.” —Deputy Director for Curriculum and Learning, Special Education and Special Services Directorate, MOEC have not been socialized to inclusive education. In a few instances personnel are not exploiting opportunities for funding. Paper. Before implementation of the OVC program, the Government of Indonesia had established laws that covered inclusion and inclusive education. Article 31 of Indonesia’s 1945 constitution states that the government should provide education for all free of charge. The GOI has also enacted three laws encompassing IE: • Law No. 4, 1997 regarding persons with disabilities states that there should be equal opportunities for people with disability in all aspects of human rights including access to education. • Law No. 23, 2002 protects the rights of children. • Law 20, 2003 regarding Indonesian education systems states in article 15 that children with disabilities can be enrolled in special schools and/or regular schools with an inclusive education. According to all government administrators whom the evaluation team interviewed, the policy work of the OVC program was extremely important in helping ensure that the laws and articles above could be extended. Participants also commented that the OVC program helped ensure that policy content and structure at provincial and district levels aligned with national policies/decrees. An important feature of OVC policy work was the development of an inclusive education policy template that allowed districts to input their information to the template to develop an inclusive education policy for their district. Most district-level evaluation participants said that OVC had contributed to their umbrella policy on inclusive education. With establishment of regulations and decrees on inclusive education and budget allocations, the participants said that they would be able to continue their efforts to implement inclusive education.24 At the national level, the advocacy work that OVC conducted with the Directorate of Special Education led to increased support of IE in 22 districts and allocation of IDR 20.7 billion (USD 2.07 million). However, one hindrance to IE was passage of the law Permenpan No. 48 (2005), which limited the number of honorary teachers that districts could hire as resource teachers. According to Nganjuk and Tuban District officials, this hampered program implementation. 24 Based on comments from education stakeholders in Aceh and East Java provinces and in Nganjuk, Pidie, and Tuban Districts. 18 | Page 0 1 2 3 4 5 District 1 Nganjuk 2 Tuban 3 Pidie 4 Lhokseumawe District Policy 0 1 2 3 4 5 6 7 8 Provincial Policy Jakarta East Java Aceh Figures 3 and 4 illustrate the number and topics of policies that stakeholders said that different levels of government had developed with OVC assistance. Figure 3: Number of Provincial-Level Inclusive Education Policies Figure 4: Number of District-Level Inclusive Education Policies by Policy Focus Area 19 | Page KEY CONCLUSIONS District-, provincial-, and national-level government personnel view the policy development initiative as the most important of the OVC initiatives. The program was highly successful in its ability to bring together a diverse group of government stakeholders from various parts of government (not just the education sector) and guide these groups to produce 22 relevant policies at different levels of the system. The effects of the policy development initiative will be long lasting because the policies promoted by the OVC program are now legislated. Furthermore, many of these policies have already resulted in concrete actions in terms of financing inclusive education; designation of schools, districts, and provinces as inclusive education sites; and changes in the government classification of personnel. The model used by OVC for the policy development initiative is one that any government sector could use to good effect. Certain aspects of the initiative, however, were lacking, Good public policy endeavors must go beyond policy drafting and incorporate policy review and policy revision; high-quality data linked with the particular policy under development, review, or revision should inform all three stages of policy development. However, these dimensions of policy development were not addressed, with implications for the quality and effectiveness of the policies produced. Essentially, the work of OVC on this initiative only comprised the first step in the policy development cycle. An analysis of policy coherence among levels of government shows it is emerging. The greatest coherence from the national to the provincial to the district levels exists among DKI Jakarta and its districts. For example, DKI Jakarta policies not only address all MOEC topical foci but also policies beyond those of MOEC. In contrast, no policy coherence exists between Lhokseumawe District and the provincial- and national-level policies and no policies have been passed by local governments in that district. All policies produced or passed during the OVC program in the three selected provinces and four selected districts, except for Lhokseumawe, will continue to be in place post-OVC, which will mean that the schools in those provinces and districts will continue to be IE schools. Also, the “grand designs” put in place with the support of OVC will continue to be used at all levels of government. GOAL 2 Improve the Capacity of University Special Education Departments KEY FINDINGS During phase 3 (N-OVC), HKI signed memoranda of understanding to establish partnerships with six universities—State University of Jakarta (UNJ), Indonesia University of Education, State University of Yogyakarta, State University of Surabaya (UNESA), State University of Padang, and Makassar State University) in five of the six targeted provinces. A university from Aceh was not included because no institution in the province to date has a special education department. The HKI-university partnerships were intended to develop the capacity of a selected subset of lecturers from each preservice special education department; the purpose was to enhance university capacity to deliver courses in one of each of the following areas: emotional intelligence, intellectual disabilities, learning disabilities, multiple disabilities, and vision impairment. To date, 19 lecturers of the selected 23 completed specialized training. Four of the lecturers participated in training at the Hilton Perkins International Institute in the United States. 20 | Page Universities and their respective faculties are accredited by the MOEC Higher Education Directorate. The universities are assessed across a fairly generic set of standards. In the case of Indonesia, the accreditation standards for special education departments include the following: • Standard 1: Vision, Mission, Goal and Target, Strategy Achievement • Standard 2: Mentoring Procedures, Leadership, Management System, and Quality Assurance • Standard 3: Students and Graduation • Standard 4: Human Resources • Standard 5: Curriculum, Learning, and Academic Climate • Standard 6: Budgeting, Facilities and Information System • Standard 7: Research, Community Services/Engagement, and Collaboration with Other Agencies OVC documentation does not show that OVC deliberately set out to align with or fall under the accreditation standards. Nonetheless, it is sensible to discuss the OVC tertiary education subsector interventions using these standards as a framework. Standards 1, 2, 6 and 7 do not represent the core areas of OVC focus and are presented in one section; Standards 3, 4 and 5 do represent core OVC foci and are discussed separately and in greater detail. STANDARDS 1, 2, 6 AND 7 These standards include (1) Vision, Mission, Goal and Target, Strategy Achievement; (2) Mentoring Procedures, Leadership, Management System, and Quality Assurance; (6) Budgeting, Facilities and Information System; (7) Research, Community Services/Engagement and Collaboration with Other Agencies. With the positive press on inclusive education and MOEC policies, all UNJ evaluation participants noted new interest among university leaders to make the university inclusive. UNJ is considering establishing a center for disability studies, and the dean has spoken with two of the special education department faculty about this idea. Furthermore, the Syiah Kuala University dean of the education faculty is interested in establishing a special education department. However, preservice education presents some challenges: • Preservice education programs are focused on preparing students to teach in segregated special education schools. • Preparation of teachers with specialized knowledge and able to support a range of children with diverse disabilities in the mainstream classroom (“pull-in”) or in a separate room in a mainstream school (“pull-out”) is not present in the preservice domain • Mainstream classroom teachers are being prepared to teach in traditional classrooms rather than in classrooms that are inclusive. Key informants mentioned physical infrastructure and assistive devices for learning—specifically, lack of accessibility for physically disabled students—as a significant problem for UNJ in general and for getting students into preservice education specifically; faculty are located several floors up and no elevator is available. Although UNJ evaluation participants said that the Jobs Access with Speech software and hardware provided through OVC is still functional and used, UNESA participants said their system was broken and had not been functional for some time. 21 | Page “We introduced the concept of multiple impairments to our students before OVC. Through OVC, we obtained a broader understanding of LD/ID and MD/VI, which provided us with more capacity to cover the subject areas with our students.” “OVC through their consultants, Dr. Jayanti Narayant and Mr.Tolhas Damanik, introduced the departments of special education to new techniques and methods to deliver MD/VI and LD/ID, which enhanced the capacity of the faculty to understand the concepts.” —Comments from UNJ and UNESA lecturers trained under OVC STANDARD 3: STUDENTS AND GRADUATION According to OVC documentation, more than 1,000 university students across the six universities have participated in the newly created or enhanced existing courses. Some anecdotal evidence exists that more students with disabilities have been accepted into the universities and awareness is at least emerging that universities should take steps to make campuses more friendly for students with disabilities. UNESA special education faculty noted that 40 students with disabilities are currently enrolled. A UNJ lecturer trained under OVC noted that, “Now our students [with disabilities] in the university [are] approaching 15 students and most of them are students with a visual impairment.” Some individual students with disabilities both within and outside the special education faculties have benefitted from some of the OVC-trained lecturers taking a personal interest in the students. Evaluation participants from UNJ said that before the OVC program, little sensitivity was shown toward students with disabilities enrolled or applying for admission and applicants with disabilities are often rejected, presumably on the basis of their disability. Other lecturers thought that bias and discrimination against potential students who have disabilities is present throughout the tertiary education subsector. STANDARD 4: HUMAN RESOURCES Lecturers participated in training sessions with short-term technical experts to upgrade their knowledge and skills. Evaluation participants in these sessions considered the training especially valuable and appreciated the hands-on learning opportunities. They also noted that the short-term technical experts were competent and credible and said they valued the access to a wide range of materials. All 17 of the lecturers interviewed and who had participated in the initiative noted how valuable it was to be able to interact on a professional level with lecturers from other universities. All lecturers who were directly involved with OVC were able to identify specific areas in which they believed they had gained new knowledge and skills. One UNJ lecturer described an “Aha!” moment in this way: “Before, when someone would ask me what the definition of multiple disabilities and vision impairment (MD/VI) meant, I would just call out a list of things—‘Oh, it is blah, blah, blah, blah.’ Now, after my experience at the Perkins Institute, I actually have a deeper understanding of what it feels like to be MD/VI! And, I use this understanding to change the way I describe what MD/VI is to people, to make it real for them. I also am much more empathetic toward my students and I work with them differently than I did before.” When this individual was asked if he had used the technique with his colleagues on his return, he looked somewhat surprised and replied, “No, but it would be an interesting thing to do and would help others understand also.” 22 | Page Special education department lecturers who had not participated directly in OVC activities noted that the engagement had improved the capacity of their colleagues. They also believed that the sharing of the new knowledge and skills across the respective departments was ad hoc and informal rather than intentional and strategic. STANDARD 5: CURRICULUM, LEARNING, AND ACADEMIC CLIMATE The OVC program worked with lecturers to produce IE course syllabi and other course materials including textbooks. Participants are using materials provided and developed or translated under the program. Several of the lecturers interviewed stated that OVC had been a catalyst for strengthening special education department course offerings, especially on the concepts of LD, ID, MD, and VI in their respective universities. According to UNESA Faculty of Education deputy dean for academic affairs (who was the head of the Special Education Department when OVC started in East Java), LD was made a specialized major to which students can now apply because of the university’s partnership with OVC. The other five universities continue to draw on the LD materials provided under OVC to enhance content in existing courses but have not yet created a new major. According to university staff interviewed, the approach used by the OVC program to develop a textbook on multiple disabilities and vision impairment was not particularly successful. One lecturer noted that those tasked with textbook development had no experience in writing such books and they had not received any instructional materials on how to develop the book. In addition, the writers could not access online reference materials because their universities have no access to EBSCO Information Services or a similar academic search engine. One book is stll in draft and has not been published. The content, however, is being used on an ad hoc basis; one lecturer who attended the workshop said he just pulls out bits of the content to use in his own classes. The OVC program also worked with the universities to develop the practicum portion of the special education degree. Before OVC, the preservice program curriculum was mainly theoretical in nature. UNJ and UNESA lecturers interviewed thought the practicum was a positive addition Two UNJ lecturers noted (and the other four lecturers concurred), “Since OVC introduced the practicum part for the courses on LD/ID at UNESA and MD/VI in UNJ, in the views of the lecturers, teaching these courses regarding disability has changed.” University participants also noted that the practicum presented numerous challenges, including a lack of schools for placement of preservice teachers. Additional evidence on issues faced under the initiative appears in HKI’s internal evaluation of the preservice education initiative.25 Some limited evidence exists on small changes. University evaluation participants pointed to changes in testing norms. For example, some students with disabilities have received five additional minutes during tests. KEY CONCLUSIONS Benefits to individual lecturers clearly accrued, however, the widespread application by all 23 lecturers of new knowledge and skills is not a given. The focus on individual lecturers without a corresponding “whole department” capacity-building approach and linkages with the faculties of education more broadly created issues that could have been anticipated and avoided. More engagement with university leadership would likely have avoided most of the weaknesses that emerged during this initiative and provided greater impact and sustainability. 25 Helen Keller International (2013), Evaluation of the OVC Program Pre-Service Education (Jakarta, Indonesia). 23 | Page “I have learned to be much more patient with children with disabilities, especially with the children who are slow learners or who are LD/ID. I used to get so frustrated and mad that I might beat the child. I don’t do that anymore. I now know that the child is not necessarily being naughty or bad.” —Resource teacher working in an inclusive education satellite school Unresolved challenges remain for the MOEC Directorate General for Higher Education to pursue: these include building on initial individual capacity development toward a “whole department” effort, “inclusivizing” faculties of education overall, identifying and funding inclusive education research, and assisting Syiah Kuala University in establishing a special education department to save on costs associated with Aceh teachers being sent to the Indonesia University of Education (Bandung). These challenges may in fact serve as a catalyst in getting this MOEC unit engaged. In addition, the transfer of knowledge and skills to other special education department colleagues and extension of knowledge and skills to “inclusivize” faculties of education more broadly remains ad hoc, informal, and limited and needs to be exploited more fully to gain maximum benefit from the OVC university work. GOAL 3 Improve In-Service Training Programs KEY FINDINGS The OVC program introduced a model consisting of five phases of professional development. These phases were delivered in a few days in several successive months. Topics ranged from basic knowledge about disabilities to specific teaching methods for children with different types of disabilities. Clear evidence exists on the widespread continuation of the five phases model; however, the phases have been scaled down and time reduced. Only one education administrator from the evaluation provinces and districts said that his office was trying to stay with the five phases (see box). Individuals participated in some or most of the five phases of in-service training and 91 teachers completed all five phases for becoming resource teachers. Most of the resource and mainstream teachers interviewed noted changes in their level of understanding of children with disabilities. Some teachers said that their professional knowledge had increased and gave examples of new information they had learned; however, they also noted that they did not use this knowledge and new skills regularly. According to a mainstream classroom teacher in an inclusive education model school, “We have changed the way we report. Now, we make two reports for children with disabilities—one is the academic progress report using the KKM minimum achievement indicators, although we use lower levels of the indicators for children with disabilities. The other one is a descriptive report that talks about things like motor skills.” Five teachers in the FGDs made similar remarks. “HKI had a huge contribution in developing the five phased training model for teachers and other stakeholders in my province. I promise to continue this model for future IE training programs.” —Head of Special Education Section, East Java Provincial Education Department 24 | Page All participants in the 24 FGDs with district leaders, school leaders and teachers (resource teachers/trainers/mentors, and mainstream teachers), all provincial and national-level education administrators, and three development partners said that the lack of capacity in the classroom and school to accommodate children with disabilities effectively is the main issue facing the sector on inclusion. One resource teacher at a model inclusive education school stated, “We learned how to develop specific learning materials. But most teachers don’t use different methods or materials for children with disabilities. Even if they participate in a training, they don’t change their teaching method.” This opinion was expressed in different ways by most teachers interviewed. This finding mirrors that of the USAID policy environment study, which found the following:26 Sixteen district education offices under MOEC confirmed that they have schools which provide inclusive education within their area, 10 officers said they need technical capacity/training on inclusive education, three on the operational funds and one on the infrastructure for inclusive schools. Of the 25 school providers of inclusive education in line with the Ministry of Education regulations, 10 schools reportedly require strategy/training on inclusive education and one school requires operational funds. Regarding the need for capacity building for the MORA district office, seven of them said they needed technical/training on inclusive education, one requires additional operational funds, and two require additional infrastructure for inclusive schools. A number of challenges arose in OVC teacher training: • The need to become proficient in sign language was challenging for resource teachers according to OVC program documentation. • In Aceh, teachers’ current level of knowledge on basic child development and learning skills required the allocation of more resources for training than anticipated. • High turnover rates of trained staff within the provincial and district education offices meant undertaking a continuous training program to ensure new personnel understood inclusive education. Mainstream classroom teachers consistently said after participating in some training that they still felt unprepared to address the learning needs of children with disabilities and this lack of preparation made them nervous and anxious. They expressed a consistent willingness to include children with disabilities but asked for support to help them do so. KEY CONCLUSIONS The five-phase model of in-service training introduced under the OVC program has positively impacted the approach and materials available for professional development, actual professional development of in-service teachers, and the government’s post-OVC efforts. Issues with consistency of participation; however, during the OVC program, have affected the larger impact of the initiative. 26 USAID (2013), Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students, (USAID PRIORITAS) Inclusive Education Policy Study (Jakarta, Indonesia). 25 | Page “Parents are always afraid to show their child. They keep them hidden. Now, since OVC and the enrollment drive, parents are bringing their children out of hiding to school.” —Resource teacher from an inclusive education model school in Jakarta) “The main beneficiaries of the OVC program are the parents of children with disabilities. The parents do not have to hide their children anymore. Their children can do things by himself or herself. Some children with disabilities have gotten outstanding achievements in their academics that make their parents very proud of the child.” —Mainstream classroom teacher from an inclusive education satellite school in Aceh Post-OVC rollout by the government with modifications in the model due to lack of funding are symptomatic of problems with scale-up in many programs worldwide, including Indonesia, and this situation will affect quality and ultimately overall impact of the efforts. Chronic problems with a high level of turnover/transfer of trained personnel to new positions among schools will continue to degrade capacity-building efforts by donors and the government, unless straightforward solutions (such as legislation that reduces the level of turnover in personnel) are found. The size of Indonesia’s population and the number of schools, districts, and provinces prevent OVC training benefits from having a positive systemic impact: the number of people trained is insignificant compared with the number not trained. This imbalance reduces the effectiveness of the transfer of knowledge and skills from trained to untrained individuals and the inability of trained individuals to apply new skills;. GOAL 4 Increase Awareness of Inclusive Education within the Education System and the Public KEY FINDINGS The family and communities. The OVC program’s awareness raising targeted homes and communities. Thirteen radio and some television talk shows were aired across Aceh, Central Java/Yogyakarta, East Java, Jakarta, South Sulawesi, and West Java. In one OVC program area in Jakarta, awareness raising through radio and identification and enrollment drives was so successful that OVC had to cease its community awareness-raising activities because the demand created threatened to swamp the education system. It is not clear whether the government is continuing to use these radio and television programs post-OVC. Individuals in all 29 FGDs and most of the key informant interviews noted that parents of children with disabilities were not as afraid to bring their children out into the public and to school and attitudes toward children with disabilities had significantly shifted at the local community level. Mainstream teachers from four inclusive education model and satellite schools noted that “regular” students are increasingly more accepting of children with disabilities. These "regular” students help children with disabilities, for example, with schoolwork or to go to the toilet. They are making friends with them. One teacher noted that “I only know of one instance where a ‘regular’ student refused to sit next to a child with a disability.” 26 | Page “We do an assessment of the child in the first week of attendance and if we don’t think we can help the child, we advise the parent to go to the model inclusive education school or to a special education school. Sometimes, when we see that the child is blind, we tell the parents to go to the special education school since we can’t help blind children here.” —Principal from an inclusive education satellite school in East Java Personnel from at least 7 of the 38 schools visited shared many stories of parents with their child being turned away from a school due to a lack of personnel and/or facilities to accommodate the child. This was particularly the case of children who were blind and/or deaf and/or had a mobility issue. Several principals said they advised these parents to visit the district or provincial special education school. Our FGD data show a clear divide along socioeconomic status in the options open to and taken by parents. Parents who are better off financially often hire an assistant for the child so that he/she can attend the mainstream school and participate. They are also able to afford tutoring and the assistive devices needed for mobility and/or learning. In one Jakarta school visited, one classroom had three aides sitting side by side with three children with a visual impairment. This finding reflects a similar finding in the USAID inclusive education policy study.27 According to the policy environment study, principals also ask these aides to work with other children with disabilities who are from poorer families.28 Our data found no evidence of this happening, however. All 63 parents interviewed spoke about their increased knowledge and ability to interact in a more positive way with their child, although our FGD data indicate a clear difference in parents’ level of understanding that seems to correlate with greater socioeconomic status and urban compared with rural locations. Part of parents’ increased capacity to understand their child and their rights can be attributed to OVC awareness-raising work, as well as the opportunity that some parents have taken to go to school with their child to observe the way the “OVC-trained teachers” interact with their child. “As a result, these parents” have begun to imitate the teachers’ behaviors. A parent from East Jakarta said, “The most important thing is how we communicate to our children. More often, we [parents of children with disabilities] are taught by the resource teacher. We often come to school and watch these resource teachers teach our children.” The stories of insight gained that were shared by many of the parents who were less well off or educated were extremely poignant. Other parents echoed the words of a parent from Nganjuk District: “I never knew something was wrong with my child. I thought he was lazy or naughty. Now, I know that he has a problem and I have to try to help him. I even know what his problem is called.” Provincial stipends are helping some parents, who apply them toward an assessment of their child with a disability. For example, some South Jakarta parents receive IDR 350,000 (USD 35) every three months for health and IQ checks for children from the government; some East Jakarta parents receive IDR 100,000 (USD 100) annually for the same purposes. Thirty families receive scholarships each year from the Nganjuk District government to help offset such expenses. 27 Ibid. 28 Ibid. 27 | Page The education system. In each OVC province, district, and school visited, evaluation participants consistently mentioned the noticeable increases in enrollment and attendance of students with disabilities. School personnel noted that the number of children went from none or a handful of children into the double digits in a single year. For example, an Aceh school has been an inclusive school since 2008, at which point, no children with disabilities were enrolled; now, of 520 students enrolled, 36 are children with disabilities. DKI Jakarta went from 164 inclusive education schools in 2011 to 374 in 2013. As mentioned earlier in the report, evaluation participants pointed to an emerging trend of parents of children without disabilities withdrawing their children when a school was designated as an “inclusive education school” by the government. Solid evidence also exists that many (if not all) schools have children with disabilities attending, whether or not the school has been designated as an “inclusive education school” by the government. In 11 secular government schools and faith-based public and private schools (not designated as inclusive education schools) in various parts of the country, a 2006 World Bank study found that of 2,975 children enrolled, 496 were identified as having a disability. At the school community level, the evaluation team carried out 33 structured school environment assessments using a school environment checklist29 and analyzed the data according to two broad categories: school grounds and learning environment. Although the checklist was not exhaustive from a disability-sensitivity perspective and the team was generous in how they assessed the schools, the data show that across the 33 schools, none of the schools had a very disability-friendly learning environment and only a handful of schools had school grounds that were very disability friendly. The evaluation team also assessed whether some of the schools visited used insulation in their roofs; none did. Figure 5 breaks down the checklist ratings according to categories by school type. Although our evaluation team attempted to identify differences and similiarities in the treatment of inclusive education by secular and religious schools, no substantive data emerged on which to base any findings. Many of the schools visited (including the 33 schools in which the checklist assessment was undertaken and 5 schools visited but not formally assessed) were undertaking construction programs or had undertaken some infrastructure upgrading to make the school disability friendly and had left the job unfinished. MOEC, USAID, and the World Bank all have construction standards for schools. None of the schools visited by the evaluation team conformed to the minimum MOEC standards or showed evidence of attempts to use “universal design” principles that make the school fit for multiple functions and purposes. International research shows that it is significantly more costly to make an existing structure disability friendly than to ensure that the structure complies with these design principles from the start of construction. 29 The School Environment Checklist was based on the OVC school checklist in the hope that the evaluation team could compare its data with that of OVC to identify any improvements or degradation at the school community level on disability sensitivity. Unfortunately, this comparison was not possible due to issues in OVC monitoring and evaluation, including the absence of baselines. 28 | Page Figure 5: Inclusive Education Friendliness by School Type According to the United Nations International Children’s Education Fund’s (UNICEF’s) The State of the World’s Children 2013: Children with Disabilities, the cost of integrating accessibility into new buildings and infrastructure amounts to less than 1% of the capital development cost; whereas adaptations to completed buildings (especially smaller buildings) can reach 20% of the original cost.30 KEY CONCLUSIONS The OVC program has reduced the invisibility of children with disabilities. Data from the evaluation show that participants believe a significant shift has taken place at all levels of government and in communities and families on the meaning of inclusive education and the possibility that children with disabilities can actually participate as students in mainstream schools. The broad consensus across evaluation participants, increases in enrollment of children with disabilities, and anecdotal evidence of parents at least trying to enroll their child31 indicate that there has been substantive benefit derived at the very local level from the awareness-raising efforts. It is not clear whether the government is continuing to use the radio and television programs produced to raise awareness on children with disabilities nor how the success of the information campaign is being measured. The absence of a clear metric for gauging the impact of the awareness raising is regrettable. The high-visibility activities of OVC, including the radio and television programs, enrollment drives, and recently, the National Inclusive Education Awards, appear to be the main reasons for this shift, at least within the general public. The shift in consciousness experienced by government personnel appears to come from continuous and persistant advocacy by OVC personnel, the awareness-raising training, the workshops to develop policy, as well as the Inclusive Education Awards. Figures cited by stakeholders about increases in enrollment of children with disabilities also suggest a shift in the 30 UNICEF (2013), State of the World’s Children 2013: Children with Disabilities (New York, NY). Also see USAID (2013), The Nexus of Orphans and Vulnerable Children, HIV and AIDS, and Equitable Access to Education (Washington, DC: USAID Orphans and Vulnerable Children Technical Working Group). 31 Although in a number of cases cited by interviewees, these children were turned away. 1 4 1 0 0 2 3 1 1 4 2 0 0 6 1 0 0 8 0 0 0 3 5 0 0 6 4 2 0 1 9 2 0 1 2 3 4 5 6 7 8 9 10 VF MF SF NF VF MF SF NF School Ground Learning Environment Special Education School Cluster Main/IE Model School Cluster IE Satellite School Cluster Satellite School VF: Very Friendly MF: Moderately SF: Somewhat NF: Not Friendly 29 | Page educational enabling environment for children with disabilities. However, without a set of established criteria and baseline data against which to gauge the inclusive educational environment, it is difficult to empirically establish shifts toward inclusiveness. Although this trend toward improved understanding and consciousness on inclusive education is obviously positive on one level, it also brings about significant challenges to the system’s capacity to serve all children, which are as yet unmet. Even though parents of children with disabilities are benefitting from improved knowledge and permitting their children to participate in mainstream education, the families whose children with disabilities who are getting into, staying in, and succeeding in school are are highly likely to be those who can afford to help their child. The apparent lack of coherence and cohesion between MOEC and MORA at the national level has also had repercussions regionally and locally that negatively affect the paradigm shift from a traditional to an inclusive education system. It is difficult, however, to identify any specific differences between secular and religious schools in this regard from our data, so this area merits further investigation. The Analytical and Capacity Development Partnership, funded by several donor agencies, has undertaken interesting comparisons of the competency of government secular school and madrassah principals, which provides important information.32 The center-to-periphery approach used in the cluster system, described in the introduction, may be effective for certain types of objectives and activities such as awareness raising and very basic skills development, but the approach is inadequate for widespread school change in general and in the classroom specifically. The schools in our evaluation study have been unable to adequately accommodate the children with disabilities who are currently enrolled and attending, unless the child comes from a family that can afford to subsidize the school’s services and pay for medical and developmental interventions that enable the child to learn. The schools visited are also clearly not ready to accommodate any increases in numbers of children with disabilities. There are widespread issues that affect all children including unsafe school grounds and inadequate infrastructure that are not conducive to learning or disability friendly. Classroom characteristics are largely not conducive to learning by children already enrolled much less an expanded population of learners. Construction standards tend to deal with the most obvious elements linked with disability friendliness, such as ramps. Less obvious priorities include window dimensions and skylights so that everyone in the classroom can see easily, and insulation in roofs (especially tin roofs) to cut down on noise generated by children and and rainstorms and to provide better heating and cooling; these are integrally linked with the quality of instruction and learning. Mainstream teachers are only just beginning to use modern teaching methods with the general population of students, let alone adapting teaching methods to accommodate a broad range of learners, including those who need assistive devices. They are also just beginning to learn and use specific types of relevant teaching and classroom and student management techniques. There is a significant risk if the school environment is not able to accommodate children with specific learning and infrastructure needs and if the government continues to designate schools as “inclusive education” schools. These inclusive education schools may, as a result, shift from being fairly 32 Analytical and Capacity Development Partnership (2013), School and Madrassah Principals and Supervisors Competency Baseline Study: Report on the Findings (Jakarta, Indonesia: MOEC). Retrieved from www.acdp￾indonesia.org/news/detail/129/1/ministry-of-religious-affairs-and-acdp-coordination-meeting-. 30 | Page “We need more support! We went from 2,000 students with disabilities in 2011 to 8,000 students in 2013 in East Java alone! Commitment only can be achieved in three years of implementation, but not the depth needed to sustain change because this inclusive education is a very new thing.” —East Java Provincial Department of Education administrator functional in serving the general student population to becoming “ghettoes” for children with disabilities and those children whose parents cannot afford to move the child to a “noninclusive education” or “regular” school. Designating a school as inclusive is not enough. Government needs to make inclusive education at the school level highly desirable and sought after by parents of children without disabilities who need to see these schools as special, unique, and desirable because they give the “normal” child an education that is beyond what she/he could get anyplace else. DISABILITY EQUALITY, SENSITIVITY TO GENDER, SEXUALITY, AND GENDER-BASED VIOLENCE DISABILITY EQUALITY IN THE SCHOOL KEY FINDINGS OVC introduced the “identification, assessment, enrollment” construct. Most government evaluation participants were familiar with the construct, and some participants, particularly those who are resource teachers and principals, have applied the construct from start to finish and used the guide on the characteristics of various disability types that was produced under OVC. As noted elsewhere in this report, the enrollment drive has been particularly successful at bringing children out of hiding and into the system. The government provides some financial assistance to families that have children with disabilities. No data are available to indicate whether this assistance is going to the most needy and exactly how it is being used. Our data indicate a difference between Jakarta and other locations regarding availability of and access to highly trained professionals in general and those who can provide services such as intelligence testing and testing for learning disabilities, and in particular, diagnosis of location on the autism spectrum. OVC phase 1 data indicate that, of 4,700 children in DKI Jakarta who were tested for vision, 600 needed glasses and 147 needed eye surgery. This means that just more than 5% of the children tested needed some type of assistance to learn effectively. In many low-income countries only 5%–15% of children who need assistive technology can obtain it. The costs of such technology can be prohibitive, especially for children who need their devices replaced or adjusted as they grow. Access to assistive technology and any other specialized support children need to facilitate their interaction and participation in school should be free and available to all.33 Our evaluation data also indicate that understanding of certain disability types varies among staff and parents. This finding is best illustrated by the terminology used across the OVC evaluation sites and in government, university, and school documents. For example, at the school level, a child who is not succeeding academically is referred to as a “slow learner” or as having “learning difficulties,” “learning disabilities,” “mild retardation,” “moderate retardation,” or “temporary intellectual problem.” The 33 UNICEF (2013), State of the World’s Children Report 2013: Children with Disabilities (New York, NY). 31 | Page evaluation data indicate that, at least in OVC program locations, inequality in access depends on the type of disability a child has, regardless of the child’s sex (figure 6). Both male and female children categorized as having a learning disability or an intellectual disability34 are enrolled in and attending mainstream schools in much greater numbers than children categorized as having other types of disabilities. These findings are supported by an OVC internal impact evaluation35 and a recent study of the inclusive education policy environment in Indonesia.36 Figure 6: Numbers of Boys and Girls with Specific Disability Labels Enrolled in School Ten principals of the 38 schools visited, when asked about the types of disabilities of children entering their schools, noted a big increase in children with autism. The head of a foundation school (yayasan) in Aceh said, “Before, we didn’t accept children with autism. But the demand is so high that now we have started to accept children with autism, too. It is mainly boys who come with autism.” The school now has many students who are classified as autistic. KEY CONCLUSIONS There are clear instances of inequality within disability itself: children categorized as learning disabled appear to be overrepresented and children from other disability groups are underrepresented. The “identification, assessment, enrollment” construct is helpful, especially for identifying children with 34 In districts and schools, “learning disability” and “intellectual disability” are being paired to describe a child’s condition. This is problematic because the two constructs do not mean the same things, do not necessarily or always co-occur, and differ in the appropriate teaching and learning practices to use. Consequently, an individualized education plan will be flawed if the exact characteristics of the child’s special needs are not clearly identified. 35 Helen Keller International (2013), “Evaluation of the OVC Program Pre-Service Education Program” (draft), (Jakarta, Indonesia: HKI/Indonesia). 36 Government of Indonesia (2010), Household Census 2010 (Jakarta, Indonesia: GOI National Bureau of Statistics). 17 5 4 4 1 1 1 1 0 0 0 10 0 4 0 2 1 0 0 1 1 1 0 2 4 6 8 10 12 14 16 18 Number of Children with Disabilities by Sex and Disability Type Male Female 32 | Page disabilities who are out of school and getting them enrolled; however, it is only a first step in what should be a very specific and highly structured approach to assessment. In addition, identification may not be fail-safe, and both misidentification, missed identification, and/or misassessment can be extremely detrimental to a child. The “identification, assessment and enrollment” approach does not extend to the realm of “achievement” for the child. Without an ensuing emphasis on this dimension, children with disabilities who enroll will fill seats but not learn a great deal. SENSITIVITY TO GENDER, SEXUALITY, AND GENDER-BASED VIOLENCE GENDER EQUALITY KEY FINDINGS According to UNICEF’s State of the World’s Children 2013: Children with Disabilities, “Children with disabilities encounter different forms of exclusion and are affected by them to varying degrees depending on the type of disability they have, where they live, and the culture or class to which they belong. Gender is also a crucial factor in exclusion. Girls with disabilities are also less likely to get an education, receive vocational training, or find employment than are boys with disabilities or girls without disabilities.”37 Our quantitative data support this global finding and indicate large differences in enrollment figures between boys with disabilities and girls with disabilities (figure 7). Given the evaluation study’s limited quantitative data on disability disaggregated by sex and the lack of data collected and held by the government on these two variables, it is not possible to identify any sex-specific trends by disability regarding enrollment. For example, we cannot point to any differences in enrollment rates of boys and girls categorized as having autism or differences in enrollment rates in boys and girls categorized as hearing impaired (low vision or blindness). The evaluation primary data show that special education schools and the inclusive education satellite schools have near gender parity; whereas the inclusive education model schools and the mainstream satellite schools have significant gender disparity in the student population. However, the evaluation data do not reveal possible reasons for these differences. USAID’s recent gender policy provides guidance on gender and undertaking gender analyses to inform the program cycle; this policy38 is meant to underpin all mission and program activities. USAID’s Gender Equality Framework is intended to help program managers address the following four dimensions of equality when designing programs: equality of access, equality in the learning process, equality of educational outcomes, and equality of external results.39 The Government of Indonesia has a dedicated Ministry of Women’s Affairs to oversee integration of gender within and across ministries, and every education unit has an appointed gender focal point. The Ministry of People’s Welfare is the secretariat for an intersectoral effort to address children’s education, health, and social welfare and is advancing coordination across these sectors for boys and girls. 37 UNICEF (2013), State of the World’s Children 2013: Children with Disabilities (New York, NY). 38 USAID (2009), An Assessment of USAID’s Programs and Policies to Improve the Lives of Women and Girls (Washington, DC: USAID); USAID (2012), Gender Equality and Women’s Empowerment Policy (Washington, DC: USAID). 39 USAID (2008), Gender Equality Framework (Washington, DC). 33 | Page Figure 7: Enrollment of Children with Disabilities by Sex and School Type KEY CONCLUSIONS The treatment of gender as a specific area of inquiry and attention is not obvious in OVC documentation or program initiatives, which this is a clear weakness not only of the program but of the government and USAID in addressing inclusive education. Gender has been a focus of attention for the past several decades in international development and there are significant lessons on which to draw. There is really no excuse for the lack of treatment of this issue under OVC, especially given that girls with disabilities from poor and minority group families are among the most marginalized people in virtually any country. There will be no way to reach the government’s Education for All and Millennium Development Goal aspirations without applying a gender lens to inclusive education and undertaking serious gender analyses that look at the micro, intermediate, and macro levels. The access of children to inclusive education varies with the type of disability, which needs further examination. Gender equality is also linked inextricably to the overall welfare of a child with a disability. The education system is ill equipped to handle abuse in the general student population, let alone exposing and dealing with the various forms of abuse of children with disabilities and providing services that are sensitive to the sex, gender identity, and disability of the child. 533 116 40 55 510 55 37 14 0 100 200 300 400 500 600 Special Education School Cluster Main/Model IE School Cluster Satellite IE School Cluster Satellite School Number of Children with Disabilities Enrolled by Sex and by School Type Male Children with Disability Female Children with Disability 34 | Page REPRODUCTIVE HEALTH AND SEXUAL AND GENDER-BASED VIOLENCE KEY FINDINGS Several evaluation participants mentioned that some learners with disabilities are physically intimate with other individuals and noted that these learners often do not know that they might be doing something wrong or something that might harm them. At least five of the evaluation participants mentioned that children with disabilities watch “blue movies” (pornography) that their parents have downloaded and that children themselves often download and bring pornographic videos to school on their phones where the content is viewed by other children. These participants thought there was a danger that children would try out the acts they had seen on the videos.They noted that their schools had in place a reproductive health curriculum, but children with disabilities were often not taught the content. These data reflect those of UNICEF: “Children and young people with disabilities have been almost entirely overlooked in sexual and reproductive health and HIV/AIDS programs, as they are often incorrectly believed to be sexually inactive, unlikely to use substances, and at less risk of violence than their peers without disabilities.”40 In our evaluation study, participants in all 29 FGDs and 7 participants in key informant interviews agreed that children with disabilities are at greater risk of abuse or violence than children without disabilities. In the FGDs, 31 specific instances of abuse perpetrated by parents, community members, education personnel, and other children were described. Of this figure, participants mentioned verbal abuse nine times, saying that children with any type of disability could be the target. Participants thought girls with intellectual disabilities and/or hearing impairment were at even higher risk of sexual abuse. Anecdotal evidence from our evaluation study is consistent with international research that also points out that children with disabilities are at greater risk from violence than children without disabilities, although the magnitude of the problem is not known.41 “The reasons for this difference include societal stigma and discrimination, negative traditional beliefs and ignorance within communities, lack of social support for carers, type of impairment (e.g., communication difficulties), and heightened vulnerability as a result of the need for increased care, including medical attention.”42 40 UNICEF (2013), State of the World’s Children Report 2013: Children with Disabilities (New York, NY). 41 Hughes, Karen, Mark A. Bellis, Lisa Jones, Sara Wood, Geoff Bates, Lindsay Eckley, Ellie McCoy, Christopher Mikton, Tom Shakespeare, and Alana Officer (2012), “Prevalence and Risk of Violence against Children with Disabilities: A Systematic Review and Meta-Analysis of Observational Studies.” The Lancet, Vol. 380, September. Retrieved from www.thelancet.com. 42 Ibid. An estimated 53,000 children ages 0–17 were murdered in 2003 and about 150 million girls and 73 million boys are believed to have been sexually abused. “We just brought a young girl to the police station. It seems she was sexually violated. We also took another girl the other month to the hospital and found that she is pregnant.” —from comments made by two school administrators in a school in Aceh 35 | Page A recent study in the United States found that “children with disabilities are three times more likely than children without them to be victims of sexual abuse, and the likelihood is even higher for children with certain types of disabilities, such as intellectual or mental health disabilities.”43 In addition, district leaders in three FGDs made some mention of children with disabilities causing physical harm to other children. KEY CONCLUSIONS The lack of access to appropriate information about reproductive health leaves children with disabilities at risk of difficult situations and consequences that could be avoided. In addition, the phenomenon of the higher levels of abuse of children (and adults) with disabilities and children out of school in general is insufficiently recognized, researched, or addressed in Indonesia and internationally.44 As the population of children with disabilities (and other special education needs) in Indonesian schools continues to expand, issues of child abuse will become more visible. Schools themselves can often be the places where abuse is perpetrated, but schools are also the places where abuse of a child in the home or the neighborhood as well as the school is likely to be identified. At present, few avenues are available to address child abuse in general and even fewer structures in place to prevent and address the abuse of children with disabilities. Although OVC did try to improve the appropriate management of boarding schools so that these settings could be safer spaces, overall, abuse of boys and girls with disabilities was not an overt focus. Abuse can negatively affect children’s development and consequently their success in school and in their later lives unless serious action is taken to address the problem. SUSTAINABILITY The striking thing about the OVC program is the extent of the evidence for the sustainabilty of many of the ideas and processes introduced. Review of program documentation, interviews with evaluation participants, and the evaluation team’s direct observation in the field show that little of what OVC has introduced is not being carried forward in some manner by all levels of government. Many of the OVC initiatives are showing their staying power and likely will continue to do so, although in some cases, in somewhat altered configurations. It is one thing to get policies and structures in place; it is a another to make sure that enabling elements really do enable a child to succeed and that decision makers in the education sector ensure the education sector is capable of providing necessary support to those at the forefront—the teachers and principals. Although the “Grand Design” initiative had merit as a part of a sustainability strategy, it does not go far enough and gaps in costing for various elements of OVC inhibit uptake of inclusive education at local government levels. The absence of risk analyses and mitigation strategies to cope with predictable dimensions of the Indonesian context will continue to weaken the continuation and deepening of the good work undertaken in the respective program locales. Government scale-up and rollout to new locales post-OVC will also be adversely affected. 43 Smith, Nancy and Sandra Harrell (2013), Sexual Abuse of Children with Disabilities: A National Snapshot (New York, NY: Vera Institute of Justice). Retrieved from www.vera.org/pubs/sexual-abuse-children-with￾disabilities. 44 United Nations (2005), UN Secretary General’s Report on Violence against Children: United Nations Thematic Group on Violence against Disabled Children: Findings and Recommendations (New York, NY). 36 | Page The evidence base for our conclusions on specific OVC and government collaborative initiatives on sustainability are embedded within various sections of the report. For the sake of brevity, the key findings will not be repeated here. Our conclusions are summarized as follows: • All policies produced or passed during OVC collaboration with MOEC, the provinces, and districts are now part of the body of Indonesian legislation. Consequently, these provinces, districts, and designated schools will continue to be considered friendly to inclusive education. The “Grand Design” process put in place with the support of OVC will likely continue to be used at all levels of government, provided it has a champion and funding to advance it. • The National Inclusive Education Awards have been embraced by and incorporated into the government at the national level. It will probably not be surprising in the near future to see provinces and districts and schools enacting their own Inclusive Education Awards. • The five-phase teacher capacity development model will still be used, although the model is being shortened due to financial constraints and the desire to train as many school personnel as possible. • The identification-assessment-enrollment construct introduced by OVC will probably be used to some extent by a few individuals. However, the weaknesses inherent in this construct jeopardize its viability for the reasons discussed earlier in this report. • The university lecturers will retain their enhanced knowledge and skills; however, the extent to which the knowledge and skills are applied to greater effect in the tertiary subsector will likely remain limited without a champion and some external assistance. The lack of access by academics to external resources (via the Internet or face to face) will continue to be a barrier to strengthening the pre-service education programs. • The increased level of awareness on inclusive education and understanding of rights and so forth will at least remain with individuals who have been in contact with the OVC program. The extent to which a critical mass of individuals who are advocates for and allies of inclusive education will be built will depend on government will and responsiveness and the strength of civil society to demand change. • Although individual principals and teachers will retain their new knowledge and possibly their new skill sets, the turnover of these staff members, as well as other key government decision makers, is already eroding the benefits accruing at the school level. It is highly likely that this erosion will continue, even if leadership at the top (MOEC, provincial, and district governments) remains strong. • The enhanced networks that cross sector siloes will remain, certainly on a personal level, and possibly on an institutional level. At this point, the viability of these networks appears solid and they seem likely to continue (barring significant disruptions due to political change). • The extensive materials developed through OVC will remain available. The equipment provided through OVC such as the Jobs Access with Speech software and hardware will remain, although the maintenance is already proving to be an issue in some places and technologies quickly go out of date. Over time, as is often the case with materials and equipment, unless there is a well-entrenched and -funded plan for maintenance and uptake, these items are quite likely to eventually be locked away in dusty closets. 37 | Page PART 3: RECOMMENDATIONS LESSONS LEARNED The following lessons have emerged from the evaluation study. A separate lessons learned report was produced based on the findings and conclusions from the evaluation. It can be accessed by request from USAID/Indonesia. The content below summarizes the lessons. Lesson 1: System change can be accelerated—even by a program that has limited resources, a rolling design, and breaks between phases—provided certain factors are in place. Important factors include government leadership and commitment and a deep knowledge of the field and the local context by the implementing organization. Lesson 2: The government’s efforts to raise visibility through positive competition and opportunities to gain recognition have raised the profile of inclusive education: potential exists for doing more along these lines. Lesson 3: Inclusive education is in the very early stages of development; without immediate attention to key risks, continued success in increasing access and quality will be difficult. Lesson 4: Coordination between the MOEC and the MORA will continue to negatively impact ensuring that children with disabilities (and other special needs) succeed unless creative and committed action is taken by these two ministries. Lesson 5: The frequent turnover in competent government personnel is a significant barrier to the sustainability of government and donor investments; without immediate mitigating strategies, this instability in the system will continue to create problems. Lesson 6: Building the capacity of individual special education lecturers was effective; however, a more holistic approach that strengthened the capacity of the special education departments could have brought about greater impact and sustainability. Lesson 7: Lack of data and data integrity are significant inhibitors to sound policy development, planning, program implementation, and ultimately, student achievement. Without such data, the transformation of the existing education system into an inclusive education system will not be possible. Lesson 8: Research on inclusive education is lacking. This gap will continue to affect policy development, review, and revision as well as teaching and learning unless action is taken to generate high-quality and relevant research. Lesson 9: Children with disabilities face higher levels of abuse. This situation will negatively affect children’s education and later lives unless serious action is taken to address the problem. Lesson 10: A donor champion such as USAID and a high-quality technical collaborator such as HKI through the OVC program are important allies in inclusive education system strengthening. 38 | Page RECOMMENDATIONS FOR THE GOVERNMENT OF INDONESIA To take inclusive education development to the next level, particularly in anticipation of the end of the National Action Plan for Education for All in 2015 and the timeframe of the Grand Design Inclusive Education Strategies, the following recommendations are offered. Recommendation 1: Deepen OVC initiatives This deepening would involve building from the OVC initiatives in the following ways: • Borrow, adapt, or develop an “Inclusive Education Capacity Index” to assess provinces and districts in terms of their “readiness” for inclusive education, develop an analytical dashboard and track changes over time. This index should look at the big picture of systems development. It would be sensible to have the National Inclusive Education Task Force take a strong lead with the index and work with provincial and district inclusive education task forces. A useful model for such an index and approach is the USAID Civil Society Organizations Sustainability Index.45 Although this index is for civil society organizations, it provides a strong model that could be adapted. • Develop a “Pathway to Inclusion” template that consists of a set of structured, thematic packages that are geared toward provinces’ and/or districts’ differing levels of awareness, capacity, and commitment based on index results. For example, those provinces and districts that that are at the “new to inclusive education” end of the spectrum would receive the “Laying the Foundation” basic package. Provinces and districts where quite a bit is happening would get the “Consolidating and Expanding the Foundation” intermediate package, while those provinces and districts that are far down the path would get the “Refining the Structure” package. A key part of this activity would be to review OVC initiatives and identify ways and means of extending their impact and sustainability. The potential for doing more with the university special education departments is an obvious place to begin. Recommendation 2: Broaden the reach of inclusive education as the dominant education paradigm, rather than a “bolted-on” initiative that merely supplements the existing system by focusing on very vulnerable children. This broadening would involve the following: • Crafting and drafting a sound approach (strategic, tactical, and operational) for the Special Education Directorate to take the lead in mainstreaming inclusive education into other MOEC units, so that IE actually (and eventually) becomes the education paradigm, rather than a bolt￾on initiative focused on children with particular identified vulnerabilities. • In the future, if recommendation 2 is realized, the directorate would have worked itself out of a job—or, rather, into its real job, which is to ensure that children with disabilities and other special needs participate in a quality education. As the directorate focuses on its core tasks, a concurrent need exists to ensure that the education quality assurance cadre is able to function as a very specialized quality assurance unit for the systemwide transformation to inclusive education. 45 USAID (2013), USAID Civil Society Sustainability Index (Washington, DC: USAID). Retrieved from www.usaid.gov/europe-eurasia-civil-society/cso-sustainability-2012. 39 | Page • A sensible place to start would be to develop a mapping of all MOEC units by function and focus and identify ways and means for mainstreaming inclusive education using these existing structures. Strengthening the capacity of the education quality assurance units is one obvious starting place. Recommendation 3: Undertake cost analyses of all interventions so that the different levels of government can sensibly expand the capacity to implement inclusive education based on a clear understanding of likely costs, depending on the location. • This exercise would also help donors and development partners to identify particular aspects they might be able to support. • A sensible place to begin would be with the five phases of OVC in-service training to identify a per unit cost for an educator to go through all phases and a per phase unit costs. • Another sensible focus is the development of a unit cost for the following: a tiered diagnostic structure to get children “ready to learn.” The first stage would consist of broadly based hearing and vision testing and basic diagnostics to catch other types of disabilities for all grade 1 (and older new- to-school) students and any early childhood center students. • A second stage would consist of a provincial-level roving team of highly skilled diagnosticians who then attend to those children whose needs were not able to be addressed during the broadly based screening stage. During this specialized diagnostic stage, the diagnostic team would work its way across schools in the province to undertake assessments of children whose needs were not addressed during the broadly based screening stage, for example, children with a learning disability or autism. This stage would also ensure that children who need particular medical services (such as eye surgery) receive treatment. • A third stage would consist of a child success package stage in which the respective child has access to the specific types of resources needed to succeed. For example, a child with a hearing impairment will need annual testing, a hearing aid, and instruction in sign language and lip reading. A child with a vision impairment will need annual vision testing (unless the child is blind), new glasses annually, large print materials or a Braille machine, a cane, and adaptation of the schoolyard and buildings (some of which cost very little). Data on all of these children would be collected and maintained and fed back into the system. Recommendation 4: Identify and advance a way and means to bring to life the imperative of the district leader quoted earlier in this report: “MOEC! MORA! Join!” • Inclusive education might function as a catalyst for a substantive way for MOEC and MORA to work together (from national to local levels), especially if recommendation 3 can be advanced. A sensible place to start would be under the leadership of the Ministry of People’s Welfare in its role as the driver of intersectoral collaboration. • It would also be worthwhile to examine how MOEC and MORA collaborate in Aceh and elsewhere at the provincial and district levels and to examine some of the easy-to-resolve issues that are blocking Islamic day and boarding schools from engaging fully in existing inclusive education initiatives. Recommendation 5: Develop and implement a capacity-building strategy and operational plan to strengthen the inclusive education task forces. • Without proper resourcing, support, and guidance, there is a very real likelihood that the inclusive education task forces will have limited effectiveness. But, these task forces should ideally eventually take up the role that OVC field personnel have performed. • A sound method for monitoring and evaluating the successes of the inclusive education task forces should be part and parcel of the approach. 40 | Page Recommendation 6: Invest in strengthening research on inclusive education from macro-level studies of the quality of policies and their impacts through micro-level ethnographic research at the classroom and school levels. • As greater numbers of very vulnerable children enter mainstream classrooms, the need to understand what challenges and opportunities exist at the school level and how these are manifesting will become increasingly important. FOR USAID Recommendation 7: Contract an OVC-type follow-on program with a systems strengthening approach that builds on the OVC implementation model and incorporates a specific focus on the acquisition of reading skills by children with disabilities. • Any follow-on should use ways of working that are similar to those of the OVC program and assist the Government of Indonesia in addressing recommendations 1–3. Recommendation 8: Appraise PRIORITAS’s (and other donor education programs’) design and approach systematically and in depth using an inclusive education lens and identify any opportunities for PRIORITAS (and other donor education programs) to be grounded in the theory and practice of inclusive education. • Inclusive education goes far beyond adding a special education module to the teacher training syllabus; consequently, recommendation 8 is particularly important for several reasons:  Inclusive education is education for all; it is not an add-on to the existing system.  If PRIORITAS were “inclusivized,” it could be a key step in moving inclusive education beyond the realm of special education.  It will enable greater impact and more likely (good quality) sustainability of the OVC initiatives.  It will be an important learning exercise for USAID (agency level) and other donors in “inclusivizing” the education system of a populous middle-income developing country. • A sensible place to start would be to convene a range of the most knowledgeable advocates and practitioners of inclusive education who are intimately familiar with the OVC program to undertake a critique of PRIORITAS and identify the most likely ways to embed OVC work into PRIORITAS work. It might also be sensible to try out this appraisal process in just one province, for example, East Java, which was an OVC partner and is currently a PRIORITAS province and which has a strong inclusive education enabling environment. Recommendation 9: Draw on the extensive and high-quality work undertaken by the Orphans and Vulnerable Children initiative under the USAID President’s Emergency Response for AIDS Relief (PEPFAR) to advance a more holistic approach to addressing the needs of very vulnerable children. • Although Indonesia is not a PEPFAR-designated country, there is valuable information that can be learned from the initiative. For example, the PEPFAR orphans and vulnerable children initiative is particularly strong in advancing cross-sectoral collaboration so that the needs of the child across a range of domains (from education to health to protection, etc.) are taken into account. • The PEPFAR orphans and vulnerable children initiative is also not particularly strong in addressing the needs of children with disabilities. Recent research undertaken in four 41 | Page countries shows that these children are particularly susceptible to sexual (and other forms of) abuse and therefore are even more susceptible to HIV/AIDS infection. The link between health and nutrition and disease (such as malaria) and disability and the effects on the education system needs to be made much more explicit and public. • At the USAID level, PEPFAR has begun to explore ways and means to strengthen the nexus between HIV/AIDS and education specifically and to identify strategies to better link with and complement existing USAID education programming managed by missions’ education offices in order to be able to address the “whole of education life cycle” of a learner discussed earlier in the report, instead of just bits of that cycle.46 Recommendation 10: Produce, publish, and present a case study on the GOI and USAID OVC experience and publicize the successes of and the lessons learned through the OVC program. • The body of literature on “inclusivizing” an existing education system in a developing country is fairly limited. GOI and USAID have a unique opportunity to showcase what has been learned about transforming an education system into one in which all children are welcome and able to learn and achieve success. Showcasing the collaborative GOI and USAID effort would be a worthwhile and welcome contribution to the international literature on inclusive education (particularly inclusion of children with disabilities) in developing countries. • The journal Comparative and International Education Society and annual conference is a sensible (although not the only) place to showcase the work. 46 USAID (2013), The Nexus of Orphans and Vulnerable Children, HIV and AIDS, and Equitable Access to Education (Washington, DC: USAID Orphans and Vulnerable Children Technical Working Group). 42 | Page ANNEX 1: STANDARD DOCUMENTATION ANNEX 1.1: SCOPE OF WORK BACKGROUND Inclusive Education in Indonesia Among children with special needs, access to high quality basic education remains limited in Indonesia. Ministry of Education and Culture (MOEC) data reveal that there are around 330,000 school-age children (5-18 years old) with special needs. However, only 86,000 (26%) children with special needs are able to attend school. To support the achievement of Education for All (EFA) goals, the Government of Indonesia passed a ministerial decree in 2009 on inclusive education and education for gifted/talented children. The decree puts a strong emphasis on children with special needs having an equal opportunity for quality education services at government and private schools. In addition, raising status of the special needs unit from a sub-directorate to a directorate has helped to accelerate inclusion efforts. Resource centers supporting inclusive and child-friendly practices in schools have been created in several provinces. The central government and local governments have issued new regulations, policies, plans and programs and have made funding available to support the implementation of inclusive education in Indonesia within the context of Indonesia’s decentralized education system. Despite the GOI’s commitment to inclusive education, significant challenges remain. Poor families with special needs children are struggling to send their children to schools due to high costs. Provincial and district governments continue to struggle with the accuracy of data applicable to special needs children and the availability of qualified resource teachers. Insufficient resources and the lack of facilities to support special needs children at special and mainstream schools remains an obstacle for the successful expansion of educational opportunities for children with special needs. The Opportunities for Vulnerable Children (OVC) Program Starting in 2003, Helen Keller International/Indonesia (HKI) and USAID, in partnership with the Ministry of Education and Culture (MOEC), the DKI Jakarta Provincial Department of Education (PDOE) and over 20 local non-governmental partners, developed the Opportunities for Vulnerable Children (OVC) project, an inclusive education initiative for the improvement of access to education and the provision of support and supplemental curricula to children with special needs. This initial program (referred to as Phase 1 of the OVC program) focused on children with visual impairments in DKI Jakarta. The model, developed to be replicated for other children with special needs, employed a transparent process of inclusion and collaboration that succeeded in stimulating broad policy changes and implementing them in the school environment. 43 | Page Taking advantage of the then current policy environment and using existing resources and a strong network of partnerships, Phase 1 of this project (which ended in 2006) worked to develop the foundations of an inclusive system within DKI Jakarta by focusing on the inclusion of students with visual impairments. In 2006, HKI, USAID and MOEC decided to continue their partnership by expanding the model developed in Phase 1 to include children with other disabilities and to expand the program geographically to three new provinces: South Sulawesi, Central Java and Nanggroe Aceh Darussalam (NAD/Aceh). Best practices and lessons learned from the model developed in Phase 1 were used to refine the program objectives for Phase 2 of OVC. In 2010, HKI and USAID began implementing Phase 3 of the OVC program, New Opportunities for Vulnerable Children (N-OVC). The current phase of the project is geared towards providing technical assistance to government counterparts to sustain and improve inclusive education support systems at the national and provincial levels. Additionally, the program is building in-service and pre-service teacher training capacity to ensure that needed teaching personnel are available to manage and effectively implement necessary support services, a core component of inclusive education. By leveraging existing models in DKI Jakarta, South Sulawesi, Aceh, Central Java, West Java and East Java,1 the N-OVC program is facilitating the development of planning, budgeting, coordination, and academic systemsacross all levels of MOEC, while also helping the government to create a national plan to expand inclusive education throughout the country. Specifically, this program focuses on the following key elements:  Improved coordination of policy, planning and funding between the national, provincial and district levels;  Improved capacity of universities to provide strong practical based pre-service teaching programs to new teachers;  Improved in-service training programs implemented by MOEC for teachers currently working with children with disabilities throughout Indonesia; and  Improved awareness and of GOI inclusive education goals and Education for All (EFA) goals within the Indonesian education system and with the public. The Phase 2 OVC program evaluation (2009) concluded that OVC improved the quality of inclusive education in the target provinces. Nonetheless, program implementation was too short in the new target provinces of Aceh, Central Java and South Sulawesi to design strong programs capable of getting special needs children into mainstream schools and to provide them with appropriate services. Thus, it was recommended that the follow-on program (N-OVC) should continue working with these provinces in such areas as the development of resource centers, advocacy for inclusive education, development of the capacity of resource teachers, and expanding social marketing on inclusive education. Thus, the N-OVC program was designed to reach more beneficiaries and improve results. To meet this need, USAID provided a follow-on grant to deepen the work in the Phase II provinces. In addition, two new provinces were included (West Java and East Java) for program expansion. The current phase of the OVC program is working with MOEC and the targeted provincial governments of DKI Jakarta, West Java, Central Java/Yogyakarta, East Java, South Sulawesi and Aceh as well as their respective district governments and other relevant GOI institutions. In addition, OVC is also working with six universities (UPI Bandung, UNJ Jakarta, UNESA Surabaya, UNP Padang, UNM Makassar, and UNY Yogyakarta) to develop the curriculum, syllabi, course materials, and lecturer skill base for two special education courses: (1) Early Intervention in Multi-disability Visual Impairment (EI MDVI); and (2) Learning Difficulties and Intellectual Disabilities (LDID). 1 West Java and East Java were included in the New-OVC (Phase 3) as part of the program expansion. 44 | Page To date 13,880 children with special needs (8,630 males and 5,250 females) have been enrolled in inclusive education schools as a result of OVC’s activities. OVC has trained additional master trainers to provide training on inclusive education to the mainstream teachers in dealing with special needs children in their schools. Efforts to educate the public and increase their awareness of special needs children have been made through various media (printing, electronic, talk shows, etc.). The close partnership with MOEC, particularly with the Directorate of Special Education in Basic Education/PKLK) has resulted in more attention, appreciation and recognition of the key actors by the GOI for their contribution and achievements in support of inclusive education. In 2011 and 2012, OVC and the GOI helped teachers, principals, district heads and governors to provide wider access to quality education for special needs children. II. PURPOSE The purpose of this final performance evaluation is to assess the performance of the Opportunities for Vulnerable Children (OVC) and New Opportunities for Vulnerable Children (N-OVC) programs (both implemented by Helen Keller International) to support the government in their efforts to build an enabling inclusive environment in primary schools in Indonesia. This evaluation will measure the degree to which the program goals have been met, and the contributing factors that have been responsible for or detracted from the achievement of these goals. In addition, this evaluation should provide an initial assessment of the sustainability of program achievements and the factors that have contributed to or detracted from the sustainability of program achievements. This evaluation is intended to provide USAID, and the Indonesian government at national, provincial, and district levels with an assessment of program performance and lessons learned to inform possible future activities that promote an inclusive environment for children with special needs. The results of the evaluation will inform dialogue on creating an inclusive environment that enables children with special needs to have better access to education services. The results of the evaluation will help the USAID/Indonesia Education Office in its implementation of a major new education program, USAID PRIORITAS, which is expected to integrate inclusive education into its program components, particularly its components dealing with pre-service and in-service teacher training. PRIORITAS is expected to provide in-service training sessions on inclusive education and work with teacher training colleges to develop pre-service course syllabi on early interventions. Thus, any lessons that can be gleaned from the OVC programs may be able to contribute to the success of USAID PRIORITAS and provide useful information to MOEC, MORA and the broader community concerned about inclusive education in Indonesia. III. EVALUATION QUESTIONS The contractor shall provide evaluation services sufficient to achieve the objectives set forth above. Specifically, the evaluation should address the questions that follow: 1. What aspects of the OVC programs proved most and least effective in creating a more inclusive environment, for each of the OVC program goals? 2 2. How inclusive is the environment for children with special needs among project communities? 45 | Page 2 The OVC program seeks to (1) improve the coordination of policy, planning and funding between the national, provincial and district levels; (2) improve the capacity of universities; (3) improve in-service training programs; and (4) increase awareness of inclusive education within the education system and the public 3. What evidence is there that the project activities and results might be sustainable after the project itself is completed? 4. To what extent did the OVC programs account for and address the gender specific challenges and needs of girls and boys in their program? The questions will apply to Phase 2 of OVC as well as N-OVC (i.e., from 2006 until 2013). The contractor shall present evaluation findings to substantiate answers to these evaluation questions; findings that are based on facts, evidence, and data (all data must be sex disaggregated). Findings should be specific, concise, and supported by quantitative and qualitative information that is reliable, valid, and generalizable. Recommendations must be action-oriented, practical and specific. Recommendations should include specific actions that the program should undertake to reduce gender inequalities, if the findings conclude that gender inequalities exist In developing a response to Evaluation Question 1, the contractor should address the following issues:  The extent to which national government, provincial governments, and district governments devote additional efforts to promoting inclusive education environments, and any differences between provinces and districts.  Differences (including gender differences and use of the modules promoted by the project) in inclusive education between secular schools and madrasahs, and factors that might account for these differences.  The extent to which the program has addressed the professional development of teachers who focus on students with special needs. In developing a response to Evaluation Question 2, the contractor should address the following issues:  The extent to which community members (schools, parents, community leaders) are promoting equal access to social interactions for boys and girls with special needs within the community and beyond and whether attitudes about special needs children changed as a result of the program. Provide examples and evidence.  Differences in inclusive environments within targeted model schools and other project schools. Describe and account for any differences.  The extent to which parents get involved in the process of helping their special needs children gain access to inclusive schools. In developing a response to Evaluation Question 3, the contractor should address the following issues:  Ways the central offices of MOEC and MORA have collaborated with the program in order to institutionalize an inclusive education environment. 46 | Page  Whether OVC programs been replicated in non-project districts/schools and if so, whether practices been replicated with the same standards of quality as the original practices.  Identify any existing barriers and/or challenges to the adoption of inclusive education strategies for policy makers, educators, parents and children. IV. METHODOLOGY USAID/Indonesia does not define a full methodology for the evaluation within this RFP, but rather outlines several required components of a methodology which the offeror should build upon in the technical proposal. The contractor will examine both quantitative and qualitative approaches and a combination of secondary (existing) and primary (new) data during the course of the evaluation. Offerors are encouraged to propose additional evaluation methods, but evaluation methodology must include at a minimum the following: 1. Collection and review of existing data: Indicator Data: USAID will provide the contractor with OVC program indicator data. The contractor must determine what other kinds of indicator data are relevant and collect it (sources could include MOEC, Helen Keller International, IDP Norway, AusAID, other local organizations, etc.). Document Review: USAID will provide the contractor with all available documentation from the OVC and NOVC program (to include gender analysis, solicitation documents, award document, project Performance Management Plans, quarterly and annual reports, and relevant studies conducted by the implementer). The contractor must also review program documentation for USAID’s current basic education program. The contractor is also expected to conduct research on inclusive education in Indonesia, including policies, other programs, and other relevant research. 2. Collection and review of new data: Key Informant Interviews: Key informant interviews and/or focus group discussions should be included; informants must be equal numbers of males and females. These interviews and discussions should include, at a minimum:  USAID/Indonesia Education team members,  Helen Keller International and local partners,  MOEC and MORA officials at the Special Education Office of the Basic Education Directorate,  Heads of provincial and district education Offices (maximum 3 provinces, and 3 districts within each province), 47 | Page  School principals and supervisors, teachers, training facilitators,  Students, parents, special needs students, and other girls and boys. The contractor should also meet with program leadership and staff of USAID’s current basic education program. Structured Observation: Structured observation should also be undertaken. It should include observation of a sample of mainstream and special education teachers as they present lessons to boys and to girls (if classes are segregated) specifically look at the individual education plans, the teaching methodology and supporting material used, and student-teacher interaction during the process of learning and the extent to which teachers reinforce gender stereotypes.3 Survey Work: A sample survey of the beneficiary community should be conducted. It should survey a sample of participating teachers in the inclusive education training to assess their opinions about the training quality and their understanding of inclusive education practices. Similarly a survey of the parents of special needs children and community members should be taken on their awareness of inclusive education environments. V. DELIVERABLES AND WORKPLAN The contractor shall produce the following materials:  Detailed Evaluation Methodology and Workplan: The contractor shall submit a detailed evaluation methodology that builds upon the analytical framework presented in the offeror’s proposal. The methodology must illustrate what kinds of evidence the team will use to answer each evaluation question, and clearly define methods and tools for collecting this evidence. The methodology must include samples of evaluation tools, including questionnaires and/or survey instruments. It must also include a list of proposed interviews and sites for field visits. The workplan must provide a timeline for the different stages of the evaluation work and delineate responsibilities among team members. The initial draft methodology and work plan is due 10 business days after the contract is signed and must be approved by USAID prior to commencing field work.  Draft Main Evaluation Report: USAID/Indonesia will provide written feedback within five business days.  Draft Lessons Learned Report: USAID/Indonesia will provide written feedback within five business days.  Final Main Evaluation Report: See Section XII: Requirements for Submission of Reports for more details.  Final Lessons Learned Report in English and Bahasa Indonesia: See Section XII: Requirements for Submission of Reports for more details.  A draft press release in English and Bahasa Indonesia, to be provided to USAID, outlining key findings from the Lessons Learned Report.  Dissemination Workshop: A half-day presentation for 50-60 people to discuss the evaluation findings open to USAID, MOEC and MORA, the implementing partner staff, relevant organizations and institutions, and CSOs concerned with inclusive education. Electronic copies of the main evaluation report and Lessons Learned Report should be submitted within three months of completion to the Development Experience Clearinghouse at http://dec.usaid.gov. In 48 | Page addition, all performance data used in the evaluation should be presented, in an organized and electronic manner, to the USAID/Indonesia Program and Education and Offices where it will be warehoused for future use. 3 Disabled boys and girls may have desires to perform socially ascribed roles as fathers and mothers, but society and institutions may communicate that this is inappropriate based on their disability status. Disabled girls and women may be encouraged to undertake employment in handicrafts, dressmaking and carpet weaving skills that are time consuming, low paying and offer little opportunity for financial independence or advancement. Because these skills produce non-essential items or items for the tourist trade, these are the first markets to slump when economic times become hard. The following illustrative work plan time frame for the evaluation is presented for illustrative purposes only, as it is recognized that the contractor will develop the actual time frame in consultation with USAID/Indonesia as part of the implementation of the evaluation. It is estimated that the evaluation and the production of the requested reports will require approximately ten weeks. Time Activity Week 1 Evaluation Team (ET) reviews documents forwarded by USAID. Week 2 The ET meets with USAID, MOEC and MORA, and reviews OVC data and results. The ET finalizes its research instruments and approach. This review might be supplemented Weeks 3-5 The ET undertakes data collection in the field. Week 6 The ET analyzes data, develops findings and initial conclusions and recommendations. Week 7-8 The ET presents a summary of initial findings to USAID to receive feedback and continues drafting the main evaluation report. ET submits first draft of the main evaluation report and presents findings orally to USAID. Sample press release is submitted to USAID for review. ET makes half-day presentation on the findings, conclusion, and recommendations to USAID, MOEC and MORA, implementing partners, and broader community. Week 9-10 The ET begins work on the Lessons Learned Report and receives feedback from USAID, prepares sample press releases and other publicity materials. ET submits final draft of the main evaluation report and Lesson Learned Report. ET makes arrangements to print and distribute the Lessons Learned Report. ET finalizes the Lessons Learned Report and publicity materials and submits them to interested parties within MOEC, donors, and relevant communities. International members of the team depart Indonesia. 49 | Page VI. COMPOSITION OF THE EVALUATION TEAM There are two key positions outlined in this RFP, the Evaluation Team Leader and the Evaluation Expert. Offerors must provide CVs for these positions. Beyond these two key positions the offeror is encouraged to propose any kind of personnel structure deemed appropriate to conduct the work outlined in this SOW. Evaluation Team Leader: The Team Leader should possess graduate-level degree (Ph.D. or master’s degree, or Indonesian equivalent S3 or S2) in education, the social sciences or a related relevant field. The Team Leader should also have a minimum of five years of working experience with inclusive education projects and 10 years of working experience with basic education activities. Evaluation Expert: The Evaluation Expert should have a minimum of seven years of planning and evaluating education assistance projects. S/he must also have specific skills in evaluation methodology and planning, including demonstrated training and/or experience in developing evaluation methodologies and managing teams in primary data collection. The full composition of the evaluation team and the roles of the Evaluation Team Leader, the Evaluation Expert and other team members should be defined and delineated in the Technical Proposal. VII. PERIOD OF PERFORMANCE The starting date of this project is the signing date of the contract. It is estimated that the evaluation shall not take more than ten weeks. A six-day work week will be permitted. The contractor must submit to USAID/Indonesia copies of the Final Evaluation Report and Final Lessons Learned Reports. VIII. INSPECTION AND ACCEPTANCE OF WORK PERFORMED TO CONDUCT THE AGREED UPON PROCEDURES AND THE REPORT The work program (including detailed steps) and the draft report will be subject to approval and acceptance by USAID/Indonesia. IX. RELATIONSHIPS AND RESPONSIBILITIES The client for this award is USAID/Indonesia. The liaison for concerns arising throughout the engagement will be the USAID/Indonesia/EducationOffice. X. LOGISTICS The contractor shall be responsible for providing all logistical support, including transportation for personnel and equipment required for the completion of the assignment, work space, tech support, professional editing, etc. USAID will facilitate introductions to Government of Indonesia counterparts and implementing partners as agreed upon in the initial workplan. 50 | Page XI. LIST OF DOCUMENTS FOR EVALUATION TEAM TO REVIEW The following are a list of documents that will be forwarded to the Evaluation Team for review prior their arrival in Indonesia.  Modules, tools and training materials developed by the program  Phase 2 (2006-2009) Evaluation Report by the program (in Bahasa Indonesia)  Scope of Work of the OVC program Phase 2 (2006-2009) and Phase 3 (2010-2013)  The Agency’s recent “USAID Evaluation Policy” report and ADS 203  Indonesia’s National Education Policy related to Inclusive Education (Ministerial Decree No. 70/2009) (Bahasa Indonesia)  Selected reports by other donors addressing Indonesia’s inclusive education sector needs XII. Requirements for Submission of Reports Final Evaluation Report: Copies of the report must be prepared in both English and Bahasa Indonesia. The contractor will provide eight bound color copies of the Final Evaluation Report (four in English and four in Bahasa Indonesia), and submit PDF files for each version. The report must follow the guidelines for formatting outlined in the “How to Note: Preparing an Evaluation Report” (which also refers to the USAID Graphic Standards Manual and Sample Evaluation Report Template). The report should include the following sections: 1) Acronyms, 2) Executive Summary, 3) Background, 4) Evaluation Purpose and Evaluation Questions, 5) Evaluation Methodology and Limitations, 6) Findings, Conclusions and Recommendations,4 and 7) Annexes Annexes should include: 1) Evaluation Statement of Work,5 2) Data Collection Instruments, 3) Sources of Information (List of Persons Interviewed, Bibliography of Documents Reviewed, Databases, etc…), 4) Disclosure of Any Conflicts of Interest, 5) Statement of Differences/Dissenting Views (if applicable). The English version of the report must be professionally edited. To the extent possible the contractor must align the evaluation and reports with the “USAID Evaluation Policy.” In particular, the contactor should carefully review Section 5 entitled, “Evaluation Requirements.” One example of the many points highlighted in this section is that, where available, the evaluation should use sex-disaggregated data and incorporate attention to gender relations in all relevant areas. Final Lessons Learned Report: The report must be in Bahasa Indonesia. The contractor will provide 150 bound color copies Final Lessons Learned Report and distribute multiple copies to MOEC and MORA headquarters staff, all provincial and district education offices, and all head teachers participating in the N-OVC program, and submit a PDF file. The Final Lessons Learned Report must a) accurately 51 | Page summarize the findings in the main OVC evaluation report; b) provide practical advice, based on the OVC program results, that will further donor’s or government’s inclusive education program approaches; and c) be both easy to read and lend itself to discussion in the general media and central ministries in Indonesia. 52 | Page ANNEX 1.2: WORKPLAN 1 2 3 4 5 6 7 8 10 11 12 12 14 STAGE 1A: PRE-MOBILIZATION PREPARATION Review background documents Prepare Draft Methodology and Fieldwork Plan Recruit REDI Field Research Team Establish logistics processes at sub-national levels STAGE 1B: IN-COUNTRY FIELDWORK PREPARATION In-briefing meeting with USAID, MOEC and MORA Methodology and Work Plan Finalization Meeting with USAID, MOEC, and MORA Conduct Field Researchers' Training Conduct Province 1 Piloting (observed by REDI methodology experts, TL, IE, GRM-DW) Prepare Outlines (a) Evaluation Report and (b) Lessons Learned Report STAGE 1C: IN-COUNTRY FIELDWORK PREPARATION Develop and review instrument used for Provincial and University visit In-briefing meeting with provincial focal person for developing logical time frame for meeting Conduct FGD with Provincial Team meeting Conduct FGD with Faculty of Education Dean and Special Education Department staff STAGE 2A: IN-COUNTRY FIELD WORK NATIONAL LEVEL- JAKARTA Conduct Key Participant Interviews with USAID, HKI, MOEC, MORA Conduct Focus Group Discussion with HKI personnel Roundtable Meeting with Donor and Development Partners PROVINCE#1 Data Collection (Supervised with coaching and mentoring) Key Participant Interviews (KPI) and Focus Group Discussions (FGD) Direct Observation (DO) ) Structured School Observation (SCO) Preliminary and Ongoing Data Processing PROVINCE #2 Data Collection (Quality oversight and coaching as needed) Key Participant Interviews (KPI) and Focus Group Discussions (FGD) Direct Observation (DO) Structured School Observation (SCO) Preliminary and Ongoing Data Processing PROVINCE #3 Data Collection (Quality oversight and coaching as needed) Key Participant Interviews (KPI) and Focus Group Discussions (FGD) Direct Observation (DO) Structured School Observation (SCO) Preliminary and Ongoing Data Processing STAGE 3A: DATA ANALYSIS and PRELIMINARY FINDINGS Final Data Quality Check Data Analysis Establish Key Findings and Conclusions Establish Draft Recommendations Establish Draft Lessons Learned Discuss Draft Findings, Conclusions and Recommendations with USAID STAGE 4A: SHARING KEY FINDINGS Prepare for Stakeholders Workshop Conduct Stakeholders Workshop (.5 day) STAGE 4B: DRAFTING DOCUMENTS Prepare & Submit Draft 1 Evaluation Report Prepare & Submit Draft 1 Lessons Learned Report Prepare and Submit Draft 1 Press Release STAGE 4C: FINALIZING AND DISSEMINATING DOCUMENTS Revise Evaluation Report Draft 1 to FINAL EVALUATION REPORT Revise Press Release Draft 1 to FINAL PRESS RELEASE Translate Final Lessons Learned Report and Final Press Release into Bahasa Indonesia Submit FINAL Evaluation Report & Lessons Learned Report to USAID PHASE 2: IN COUNTRY FIELD WORK PHASE 3: MAKING GOOD SENSE OF GOOD DATA PHASE 1: PREPARATION PHASE 4: SHARING Task In-Country Weeks 53 | Page ANNEX 1.3: CONSULTATIONS FGD Participants No District Group Name 1 Nganjuk District Leader Subanuharti 2 Nganjuk District Leader Surati 3 Nganjuk District Leader Rudy M.P. 4 Nganjuk District Leader Tri Handayani 5 Nganjuk District Leader Suparji 6 Nganjuk District Leader Ruslan 7 Nganjuk District Leader Sudjiono 8 Nganjuk District Leader Abdul Aziz A. 9 Nganjuk District Leader Endang Darsiningsih 10 Nganjuk Principal, Committee, Community Mansur 11 Nganjuk Principal, Committee, Community Yatno 12 Nganjuk Principal, Committee, Community Erlin Pudjiastuti 13 Nganjuk Principal, Committee, Community Sri Utami 14 Nganjuk Principal, Committee, Community Surati 15 Nganjuk Principal, Committee, Community Marjo 16 Nganjuk Principal, Committee, Community Sujatmiko B 17 Nganjuk Principal, Committee, Community Agustin R. 18 Nganjuk Principal, Committee, Community Iswati 19 Nganjuk Principal, Committee, Community Slamet 20 Nganjuk Principal, Committee, Community Asmadi 21 Nganjuk GPK/Trainer Yuli Setiyowati 22 Nganjuk GPK/Trainer Endang Kusherliningsih 23 Nganjuk GPK/Trainer Teguh Hariyono 24 Nganjuk GPK/Trainer Bambang Susanto 25 Nganjuk GPK/Trainer Siti Umi Kulsum 26 Nganjuk GPK/Trainer Efendi Dwi Anggoro Utomo 27 Nganjuk GPK/Trainer H. Ekky Usdariawan 28 Nganjuk GPK/Trainer Sumarsih 29 Nganjuk GPK/Trainer Ninik Suprihatin 30 Nganjuk GPK/Trainer Sumiyati 31 Nganjuk Regular Teacher Esti Triwali 32 Nganjuk Regular Teacher Lilik Supriyati 33 Nganjuk Regular Teacher Dewi Kurniawati 54 | Page 34 Nganjuk Regular Teacher Niswatul Muslimah 35 Nganjuk Regular Teacher Sumiati 36 Nganjuk Regular Teacher Yuniasri 37 Nganjuk Regular Teacher Moch. Imron 38 Nganjuk Regular Teacher Zakani Ariep 39 Nganjuk Regular Teacher Ekky 40 Nganjuk Regular Teacher M. Asruri 41 Nganjuk Parents Yatinem 42 Nganjuk Parents Siti Aminah 43 Nganjuk Parents Nur Auliah 44 Nganjuk Parents Lasmi 45 Nganjuk Parents Siti Mahmudah 46 Nganjuk Parents Sinem 47 Nganjuk Parents Sukimin 48 Nganjuk Parents Pardi 49 Nganjuk Parents Salam 50 Nganjuk Parents Karni 51 Nganjuk Parents Sugono (People with Physical Disability) 52 Nganjuk Parents Misri 53 Tuban District Leader Totok Suprijanto 54 Tuban District Leader Mundi Hartono 55 Tuban District Leader Sutiah 56 Tuban District Leader Ana Muhibbudin 57 Tuban District Leader Darmoko 58 Tuban District Leader Sutarno 59 Tuban District Leader Ruddys SW 60 Tuban Principal, Committee, Community Sugeng TA 61 Tuban Principal, Committee, Community Endang PA 62 Tuban Principal, Committee, Community Solikatin 63 Tuban Principal, Committee, Community Maria Ulva 64 Tuban Principal, Committee, Community Nurhariadji 65 Tuban Principal, Committee, Community Cipta Jaya 66 Tuban Principal, Committee, Community Sari Handayani 67 Tuban Principal, Committee, Community Bambang Gunarto 68 Tuban Principal, Committee, Community Ahmad Dhofir 69 Tuban Principal, Committee, Community Teguh Sutomo 70 Tuban GPK/Trainer Andriana WH. S.Pd 71 Tuban GPK/Trainer Endah Puji W. 72 Tuban GPK/Trainer Jatini, S.Pd 73 Tuban GPK/Trainer Cholisatun N, S.Pd 55 | Page 74 Tuban GPK/Trainer M. Safi'i 75 Tuban GPK/Trainer M. Nashikhin, S.Pd 76 Tuban GPK/Trainer Wiwik. N., S.Pd 77 Tuban GPK/Trainer Dwi Tita N. 78 Tuban GPK/Trainer Meisaroh 79 Tuban Regular Teacher Ani Satul Jinan 80 Tuban Regular Teacher Rijatno 81 Tuban Regular Teacher Murni 82 Tuban Regular Teacher Afifah 83 Tuban Regular Teacher Anis Sriwahyuningsih 84 Tuban Regular Teacher Ratna Dwi kartika S 85 Tuban Regular Teacher Kismiati 86 Tuban Regular Teacher Muyasarah 87 Tuban Regular Teacher Widya Yunita Rahwulan 88 Tuban Parents Nurhadi 89 Tuban Parents Masriatun 90 Tuban Parents Angel P 91 Tuban Parents Suyari 92 Tuban Parents Siti Khusnul Khotimah 93 Tuban Parents Siti Musyarofah 94 Tuban Parents Endang Susilowati 95 South Jakarta District Leader Drs. Sutiyar 96 South Jakarta District Leader Drs. Mangatur Sinaga 97 South Jakarta District Leader Sri Mujiningsih 98 South Jakarta District Leader Renthy Evi 99 South Jakarta District Leader Rusbani 100 South Jakarta District Leader Ahmad Hilmi 101 South Jakarta District Leader Jamil 102 South Jakarta Principal, Committee, Community Hj. Haryanti, S.Pd. 103 South Jakarta Principal, Committee, Community Neneng Mulyani 104 South Jakarta Principal, Committee, Community Nuni S 105 South Jakarta Principal, Committee, Community Agus Mulia 106 South Jakarta GPK/Trainer Indri Yustianingsih, S.Pd 107 South Jakarta GPK/Trainer Dra. Satiti S 108 South Jakarta Regular Teacher Dewi Wulanjani, M.Pd. 109 South Jakarta Regular Teacher Tasnim, S.Pd 110 South Jakarta Regular Teacher Wiwi Susanti, S.Pd 111 South Jakarta Regular Teacher Inda Khofifah, S.PdI 112 South Jakarta Regular Teacher H. Akbar 113 South Jakarta Parents R. Loefti W. AR 56 | Page 114 South Jakarta Parents Rommy H 115 South Jakarta Parents Minsyahril 116 South Jakarta Parents Siti Robiatul 117 South Jakarta Parents Neneng Fitriani 118 South Jakarta Parents Sulaiman Jailani 119 South Jakarta Parents Suhartono 120 South Jakarta Parents Irwan 121 East Jakarta District Leader Suharto 122 East Jakarta District Leader Sutrisno 123 East Jakarta District Leader Lies Yuniarti 124 East Jakarta District Leader Novarina Valentika 125 East Jakarta District Leader Sunisah 126 East Jakarta District Leader Siti Sanawiyah 127 East Jakarta District Leader Danang Sudomo 128 East Jakarta District Leader Naliusmar 129 East Jakarta District Leader Umar Hasan 130 East Jakarta District Leader Kartini 131 East Jakarta Principal, Committee, Community Asto Hutapea 132 East Jakarta Principal, Committee, Community Suyadi 133 East Jakarta Principal, Committee, Community Anang Mulyana 134 East Jakarta Principal, Committee, Community Yustina Malau 135 East Jakarta Principal, Committee, Community Siti Bastiyah 136 East Jakarta Principal, Committee, Community I.G.A. Astuti 137 East Jakarta Principal, Committee, Community Sufiati 138 East Jakarta Principal, Committee, Community Lukman N 139 East Jakarta Principal, Committee, Community Nisa Rahmat 140 East Jakarta Principal, Committee, Community Sawalia Syafitri 141 East Jakarta GPK/Trainer Tsamaniatu Sitta Fitri U 142 East Jakarta GPK/Trainer Abdul Haris 143 East Jakarta GPK/Trainer Rif'atil Farihah 144 East Jakarta Regular Teacher Rikke Kumentas 145 East Jakarta Regular Teacher Elvi Gusveni 146 East Jakarta Regular Teacher Ch Sihwati 147 East Jakarta Regular Teacher Rosdiana S 148 East Jakarta Regular Teacher Sri Hartini 149 East Jakarta Regular Teacher Purnomo 150 East Jakarta Regular Teacher Yuliana Hermelin 151 East Jakarta Regular Teacher Rumondang 152 East Jakarta Regular Teacher Frenti C 153 East Jakarta Regular Teacher Enok Sopiah 57 | Page 154 East Jakarta Parents Helidey 155 East Jakarta Parents Milawati 156 East Jakarta Parents Sri Rahayu 157 East Jakarta Parents Yulius Hermeli 158 East Jakarta Parents Eko 159 East Jakarta Parents Nur 160 East Jakarta Parents Anie Lestari 161 East Jakarta Parents Nurmala 162 East Jakarta Parents Alda 163 East Jakarta Parents Dahlia 164 East Jakarta Parents Nisa 165 Lhokseumawe District Leader Hj. Nurlena, S.Pd 166 Lhokseumawe District Leader Hj. Suryana, S.Pd 167 Lhokseumawe District Leader Darhafli, S.Pd 168 Lhokseumawe District Leader M. Riyandika 169 Lhokseumawe District Leader H. Zainuddin HM Yacob 170 Lhokseumawe District Leader H. Jailani Usman, SH,MM 171 Lhokseumawe District Leader Fakhrizal 172 Lhokseumawe Principal, Committee, Community Mishbahuddin S.ag 173 Lhokseumawe Principal, Committee, Community TM Ali Yusdfi 174 Lhokseumawe Principal, Committee, Community Hj. Jarfina, S.Pd 175 Lhokseumawe Principal, Committee, Community H. Aziz 176 Lhokseumawe Principal, Committee, Community Erniati, S.Pd 177 Lhokseumawe Principal, Committee, Community Samsul Bahri, S.Pd.I 178 Lhokseumawe Principal, Committee, Community Hj. Salbiah, S.P 179 Lhokseumawe Principal, Committee, Community Sakdiah S.Pd 180 Lhokseumawe Principal, Committee, Community Gustiana Fitri, S.Pd 181 Lhokseumawe Principal, Committee, Community Chairul Zadi MZ, AP, SE 182 Lhokseumawe Regular Teacher Rosmiati 183 Lhokseumawe Regular Teacher Endang Susanti 184 Lhokseumawe Regular Teacher Nur Jamali 185 Lhokseumawe Regular Teacher Aminah 186 Lhokseumawe Regular Teacher Sawinar 187 Lhokseumawe Regular Teacher Saifullah S.Pd 188 Lhokseumawe Regular Teacher Hadiati 189 Lhokseumawe Regular Teacher Firdiana 190 Lhokseumawe Regular Teacher Rahimah 191 Lhokseumawe Regular Teacher Halimatussakdiyah, S.Pd 192 Lhokseumawe Parents Cut Eri Alfina 193 Lhokseumawe Parents Mariani 58 | Page 194 Lhokseumawe Parents Nuraini 195 Lhokseumawe Parents Zulfitria 196 Lhokseumawe Parents Cut Husniah 197 Lhokseumawe Parents Habibah 198 Lhokseumawe Parents Cut Nurjannah 199 Pidie District Leader Nasrul, S.Pd, M.Pd 200 Pidie District Leader Khairul Ibad 201 Pidie District Leader Dra. Arbayani 202 Pidie District Leader Ridwandi 203 Pidie District Leader Hasan Basri 204 Pidie District Leader Syafrida 205 Pidie District Leader f 206 Pidie Principal, Committee, Community Nursiah 207 Pidie Principal, Committee, Community Nidawati 208 Pidie Principal, Committee, Community Nazarullah 209 Pidie Principal, Committee, Community M. Nasir 210 Pidie Principal, Committee, Community Aiyub 211 Pidie Principal, Committee, Community Rusli 212 Pidie Principal, Committee, Community M. Jamil 213 Pidie Principal, Committee, Community Rosmiati 214 Pidie Principal, Committee, Community Iskandar Ismail 215 Pidie GPK/Trainer Isna 216 Pidie GPK/Trainer Rahma 217 Pidie GPK/Trainer Mardini Yanti 218 Pidie GPK/Trainer Jamaliah 219 Pidie GPK/Trainer Juariah 220 Pidie GPK/Trainer Halimah 221 Pidie GPK/Trainer Zuraida 222 Pidie GPK/Trainer Yusmarni (People with physical dissabil￾ity) 223 Pidie GPK/Trainer Nurfajar 224 Pidie Regular Teacher Raimah 225 Pidie Regular Teacher Syamsiah 226 Pidie Regular Teacher Huddiyah 227 Pidie Regular Teacher Ainol Mardhiah, S.Pd 228 Pidie Regular Teacher Azizah 229 Pidie Regular Teacher M. Ikhsan 230 Pidie Regular Teacher Zahrina 231 Pidie Regular Teacher Fakhriati 232 Pidie Regular Teacher M. Yunus 59 | Page 233 Pidie Regular Teacher Rosmini 234 Pidie Regular Teacher Nurhayati 235 Pidie Regular Teacher Fadhilah 236 Pidie Parents T. Sulaiman 237 Pidie Parents Fadliana 238 Pidie Parents Ajrina 239 Pidie Parents Ernida 240 Pidie Parents Ernawati 241 Pidie Parents Nurbaya 242 Pidie Parents Jariah 243 Pidie Parents Nur Fajriah 244 Pidie Parents Sulaiman 245 Pidie Parents Huzaimah Al Qomari, Amd. Kep. 246 Pidie Parents Erlina 247 Pidie Parents Musliadi 248 East Java Ed￾ucation De￾partment The Head of Basic education Division Drs. Nuryanto, M. si. 249 East Java The head of Special Education Section Dra. Pujihastuti 250 East Java The head of Sub Division for Planning Department (BAPPEDA) Ms. Ida Triwulandari 251 East Java Functional Planer for Planning De￾partment (BAPPEDA) Mr. Judi Aquarianto 252 East Java Inter Agency Collaboration Department (Biro Kerjasama) Mr. Adji 253 East Java State University of Surabaya (UNESA) Budiyanto, Dr. 254 East Java State University of Surabaya (UNESA) Ms. Endang Purbaningsih 255 East Java State University of Surabaya (UNESA) Dr. Sujarwanto 256 East Java State University of Surabaya (UNESA) Dr. Yuliyati 257 East Java State University of Surabaya (UNESA) Mr. Wahyudin H 258 East Java State University of Surabaya (UNESA) Dr. Zamis 259 East Java State University of Surabaya (UNESA) Mr. Madechan 260 East Java State University of Surabaya (UNESA) Ms. Asri w. 261 East Java State University of Surabaya (UNESA) Dr. S. Joeda A 262 NAD Head of Special and out of school Divi￾sion NAD Provincial Education Authori￾ty Mr. Saifullah, S. Pd., M. Pd. 263 NAD Head of out of school education sec￾tion NAD Provincial Education Authori￾ty Mr. Sulaiman 60 | Page 264 NAD Head of section for special education NAD Provincial Education Authority Mr. T. Isran 265 NAD IE task force in NAD Province Mr.Said Jufri 266 NAD IE task force in NAD Province NAD Provincial Education Authority Mr. Fadlan 267 NAD Head of Section for Teacher and edu￾cation personnel NAD MoRA M. Idris 268 NAD Dean of Faculty teacher training and Education science Dr. Djufri, M. Si. 269 NAD The head of Banda Aceh RC/Special school Mr. Syafrudin, S. Pd., MM. 270 NAD The Head of MIN Lhong Raya Ms. Sursiah, S. Ag., M. Pd. 271 NAD Deputy head teacher of MIN Lhong Raya Mr.Fadli, S. Pd. 272 NAD The head teacher of MIN Lam Rabo Mr. Iskandar, S. Ag. 273 NAD The deputy principal for student affair Ms. Haswita, S. Ag.. 274 DKI Jakarta The head of Special and out of school education division Ms. Dra. Septi Novida, M. Pd. 275 DKI Jakarta Head of section for curriculum Mr. Drs. Joko Sugianto, M. Pd. 276 DKI Jakarta Head of section for management Ms. Dra. Deasy Idawati, M. Pd. 277 DKI Jakarta IE Task force for Jakarta Ms. Dra. Sastrawati, M. Pd. 278 DKI Jakarta IE Task force for Jakarta Ms. Dra. F. Atiek 279 DKI Jakarta IE Task force for Jakarta Mr. Drs. Supardi 280 DKI Jakarta IE Task force for Jakarta Ms.Aulia Sri A 281 MoEC Director for Special Ed and Special Ed services Dr. Mudjito 282 MoEC Deputy Director for curriculum and learning material Dr. Praptono 283 UNJ Dean of Faculty of Education Dr. Sofia Hartati 284 UNJ Deputy Dean for Academic affair Dr. Gantina Komalasari 285 UNJ Head of Special Education Department Dr. Wuryani 286 UNJ Secretary of Special education De￾partment Dr. Indina Tarjiah 287 UNJ Lecturer for Special Education De￾partment Mr.Moh. Arief Taboer, S. Pd., M. Pd. 288 UNJ Lecturer for Special Education De￾partment Mr. Santoso, S. Pd., M. Pd. 289 INGO IDP-Norway Mr. Alexander Hauschild 61 | Page 290 NGO Rawinala Foundation Mr. Budi Prasojo 291 NGO Pantara Foundation Ms Henny 62 | Page ANNEX 2: METHODOLOGY TYPE OF EVALUATION USAID/Indonesia has requested an external evaluation. This means that the evaluation team consists of external evaluators. There will be no collaboration or participation in the data collection, analysis, or development of conclusions and recommendations by the Government of Indonesia, USAID/Indonesia, or Implementing Partner (HKI/I) personnel. USAID/Indonesia personnel have been provided with the Overall Work Plan for the evaluation and may meet up with the evaluation team pe￾riodically throughout the fieldwork to monitor the performance of the evaluation team. EVALUATION CONCEPTUAL FRAMEWORK The evaluation framework will involve assessment of program performance in strengthening seven key components of the system: 1. Leadership & governance 2. Well-performing workforce 3. Adequate financing 4. Information management / accountability 5. Coordination & networking 6. Service models and service delivery Use of such a framework is appropriate since the OVCP is a system-strengthening program that has grown in scope and scale over time. The various dimensions of such a framework will enable the GRM CET to “unpack” the interventions undertaken by HKI/I and discuss them in a coherent and co￾hesive manner. USAID/Indonesia is particularly interested in the extent to which the education system has been strengthened with respect to inclusive education and would like to see this orientation be￾come a key part of the Lessons Learned section of the Evaluation Report and a large part of the Stakeholders’ Meeting on November 21 at the end of the evaluation study. SAMPLING FRAME Three of the six provinces (DKI Jakarta, East Java, and NAD) and two districts and at least two schools in each district in which the OVCP was implemented will be visited, with extensive data col￾lection through FGDs, mini-FGDs, and informal meetings taking place in each of these provinces. If time and resources allow, there will also be data collection in the remaining three provinces of Cen￾tral Java, South Sulawesi, and West Java with a focus on provincial-level government personnel and university personnel. The GRM CET will also collect data from the national level from government personnel through key participant interviews (KPI) and other donors and development partners through a group meeting. It is anticipated that approximately 342–420 individuals (312–390 individuals will be interviewed in FGDs and approximately 30 individuals will be interviewed in KPIs and group meetings) will be in￾volved in the evaluation study. The three provinces were selected purposely with one province from each phase of the OVCP repre￾sented. DKI Jakarta was selected because it was the only province in Phase 1. NAD was selected because it is a unique case within Indonesia given its history of the past 30+ years and the concentra￾tion of madrassahs (Islamic religious schools). East Java was selected because of proximity and with financial considerations in mind. 63 | Page Districts were selected on the basis of HKI/I and GOI and GRM CET personnel opinions on the best￾performing district and the worst-performing district with respect to the OVCP interventions. Schools were selected randomly as will participants in the FGDs for parents, school and community leaders, special education teacher/inclusive education trainers, and regular teachers. Participants in district and provincial education leader FGDs, the key participants interviews and the group meeting with donors/development partners will be drawn according to the positions they hold. Attempts will be made to meet with half of the OVCP field staff and selection will be based on staff lo￾cation in Jakarta. Figure 1 illustrates the relative emphasis of data collection in the various OVCP target locations and at diverse levels of the education system. Figure 1: Data Collection Emphasis In all, the REDI team will conduct 30 FGDs in six districts in three provinces. FGD participants repre￾sent five different groups of individuals who were involved in some capacity with the OVCP: • District-level government personnel from the Department of Education and political leaders • School principals, school committee members, and community leaders • Special education teachers and teacher trainers • Mainstream classroom teachers • Parents The GRM Core Evaluation Team will conduct FGDs and KPis and one group meeting with the follow￾ing: • Provincial-level government personnel from the Provincial Department of Education and polit￾ical leaders (Mini-FGD x 6) • Provincial Inclusive Education Taskforce personnel (Mini-FGD x 6) • University personnel from six universities (FGD x 6) • District Inclusive Education Taskforce personnel (FGD x 3) XX National Level Key Participant Interviews XX Provincial Level Key Participant Interviews and 12 group meetings in DKI Jakarta, East Java, NAD Provinces and 3 districts XX District and School Level FGDs in 6 districts in DKI Jakarta, East Java, NAD Provinces 64 | Page • District Department of Education inclusive education focal points (FGD x 3) • School principals (mini-FGD x 3) • Special education teachers and teacher trainers (mini-FGD x 3) • Donor and development partners concerned with inclusive education (particularly, special ed￾ucation) (Group meeting x 1) • LNGO partners of the OVCP (Group meeting x 1) FGDs will consist of 8 to 10 individuals representing a range of backgrounds, experience, and qualifi￾cations. There will be a balance of females and males and every effort will be made to ensure that in￾dividuals with disabilities are part of the participant pool so that their unique perspective is brought in￾to the discussion. Mini-FGDs47 will consist of three to seven individuals selected with the same considerations listed above for the FGD participants. Key participant interview participants will be selected on the basis of position. Given the imbalance in Indonesia in representation of females in positions of authority, it will be difficult to maintain a gender balance in the KPI activity. In addition, given the limited representation of individuals with disabilities in positions of authority within the government, it will be difficult to incorporate such individuals in the KPI activity as well. It is estimated that the GRM CET will undertake approximately five KPIs at the na￾tional level with government personnel from the Ministry of Education and Culture and the Ministry of Religious Affairs A group meeting will be held with other donor and development partners in Jakarta who are involved in similar efforts to that of USAID/Indonesia and HKI and who would have a perspective on the OVCP. Approximately 10–12 individuals will be invited. A random sample of OVCP former field staff residing in and around Jakarta will be invited to a half￾day workshop to enable the GRM CET to gain the perspectives of these individuals about the accom￾plishments of and challenges for the OVCP. Approximately six individuals will be invited. During the visits to the six provinces, attempts will be made to interview the other six field personnel. It should be noted, however, that these individuals no longer work for HKI since OVCP implementation ceased around six months ago; therefore, they are under no obligation to meet with the GRM CET. METHODS The evaluation will utilize mixed methods. However, the bulk of the primary data will be qualitative and derived through qualitative research methods including FGDs, KPIs, direct observation, and group meetings. Some qualitative data will be analyzed quantitatively. These methods are based on the stakeholders to be interviewed in the field and vary according to an objective, confidential, and reliable data collection methodology. For high-level ministerial meetings with key government officials, individual interviews will be more appropriate. For interviews with NGO partner staff, families, and community elders, group interviews are believed to be more appropriate and more efficient. Secondary source documents and data (processed and raw) from the OVCP will be examined to de￾termine the extent to which these can be utilized to inform the evaluation study and to supplement the primary data being collected by the EST. 47 65 | Page TYPES AND CONDUCT OF METHODS Focus group discussions will be conducted according to established quality norms, including the following elements: • A pair of FGD researchers including one facilitator and one note taker • Written, verbatim notes of the FGD session with indications of which FGD participant made which remark • Audio recording of each FGD session • Quality check of the FGD written notes using the audio recording • Pilot testing and adjustment of the FGD protocols • Confidentiality of data • Comfort of participants and respect for any reluctance to respond to a particular question Once an FGD has been undertaken, the REDI Field Research Team will quality check the FGD notes using the audio recording and ensure that the notes are comprehensive and accurate. The REDI FRT will then analyze the responses to each of the FGD protocol questions and identify commonalities of perspectives and any differences in perspectives. These commonalities and differences will be rec￾orded in the FGD Report. The REDI Core Evaluation Team Data Specialist will combine the analyses of the REDI FRT by question and by FGD participant group into an electronic matrix. This matrix will be used at the end of the fieldwork to identify overarching commonalities and differences in perspec￾tive across the fieldwork sites and participant groups. Audio recordings (in Bahasa Indonesia) will be archived electronically. The REDI CET will ensure that all notes, FGD reports, and the matrix content are provided in hard and soft copies in both standard English and Bahasa Indonesia. The REDI Core Evaluation Team will undertake periodic quality checks during the conduct of FGDs to ensure that the FGD facilitators and notetakers are performing well. The evaluation team leader will review the written notes, FGD reports, and matrix content periodically to ensure quality and usefulness of the data and the analyses. Key participant interviews will be conducted according to established quality norms including the following elements: • Written, verbatim notes • Audio recording of interview • Quality check of notes from each KPI using the audio recording • Pilot testing and adjustment of the KPI protocols • Confidentiality of data and privacy of participant • Comfort of participants and respect for any reluctance to answer a particular question The GRM CET will ensure that all notes are provided in soft copy in standard English. Audio record￾ings will be archived electronically. Group meetings will be conducted utilizing the structured guidance and protocols for the FGDs alt￾hough the process will not conform strictly to FGD methodological norms. The GRM CET will ensure that any notes are provided in soft copy in standard English. Audio recordings of these meetings will not be produced. Direct observation will be utilized by all researchers. Researchers will observe the site where each FGD is being conducted. They will also observe the school grounds. The REDI FRT will provide struc￾tured direct observation content through the School Environment Checklist and less structured obser￾vation content in a section of the FGD reports and site, district, and province reports. Policy analysis will be undertaken using an analytical tool developed by the GRM CET. This tool will enable cross-checking of progress in the evaluation’s selected locations against the established GOI 66 | Page policies, regulations, and decrees. An analysis of policies, regulations, and decrees developed under the auspices of the OVCP will also be undertaken to examine the quality of such legislation. Communications analysis will be undertaken using an analytical tool currently in development by the GRM CET. This tool will enable an examination of the quality of the outreach efforts of the OVCP. Once the quantitative research methods specialist has had meetings with targeted media organiza￾tions, it will be possible to determine what type of analysis might be possible and sensible to under￾take. The FGDs will also supplement the examination of the community outreach through the ques￾tions that deal specifically with changes in attitudes and beliefs, knowledge, and practices of commu￾nity members. REVIEW OF SECONDARY SOURCE DATA AND DOCUMENTATION The GRM CET will review existing OVCP raw and processed data and program documentation, as well as materials produced by indirect beneficiaries, including curricula, training materials, legislation, and international literature on inclusive education systems strengthening. INSTRUMENTS FOCUS GROUP DISCUSSION PROTOCOLS Seven FGD Protocols will be utilized: • Protocol 1: Provincial Inclusive Education Taskforce FGD and District Inclusive Education Taskforce FGD • Protocol 2: University FGD • Protocol 3: Provincial Department of Education and political leaders FGD and district-level Department of Education and political leaders FGD • Protocol 4: School principals, school committee members, and community leaders FGD • Protocol 5A: Special education teachers and teacher trainers FGD • Protocol 5B: Mainstream classroom teachers FGD • Protocol 6: Parents FGD These protocols have significant commonalities in lines of questioning. Some questions that are ap￾propriate only for the respective FGD have been included in the respective protocol to draw on indi￾viduals’ particular unique areas of knowledge and expertise. KEY PARTICIPANT INTERVIEW PROTOCOL One key participant interview protocol will be used. It consists of several very broad semistructured questions, and follow-up probe questions will be drawn from the various FGD protocols, as appropri￾ate. This protocol will also be used for group meetings and mini-FGDs. SCHOOL ENVIRONMENT CHECKLIST A checklist will be applied to each school visited to gain a first-hand perspective on the individual school environment. Results from the checklists will be aggregated and a set of quantitative data provided for analysis. POLICY ANALYSIS FRAMEWORK A policy analysis framework has been developed that details the various GOI legislation pertaining to inclusive education and will be used to examine the successes and/or failures of targeted sites and organizations to implement policies, regulations, and decrees. The framework will also enable an ap￾praisal of the quality of such policies, regulations, and decrees developed with OVCP technical assis￾tance. 67 | Page COMMUNITY OUTREACH ANALYSIS FRAMEWORK A framework to investigate the efforts of the OVCP to change public and government opinions about inclusive education is under development. PREPARATION FOR FIELDWORK, CONDUCTING FIELDWORK, AND PROCESSING OF DATA PREPARATION The GRM CET will convene in Jakarta from August 24 to September 6, 2013 and undertake the tasks indicated in the detailed work plan for visit 1. These tasks include convening the REDI CET and REDI Field Research Team in Jakarta for in-depth preparation and refresher training in the conduct of an FGD as well as pilot testing FGD protocols and preparing for the subsequent REDI team and GRM CET fieldwork. CONDUCTING FIELDWORK Following the preparation phase, the following structure will be put into effect: September 9–October 21 • REDI team prepares detailed work plan and undertakes fieldwork in three provinces (DKI Ja￾karta, East Java, and NAD) and six districts. • The GRM CET inclusive education Indonesia specialist prepares a detailed work plan and undertakes desk-based research on policy and regulations and university interventions and products. The IEIS will also spend several days with the REDI CET/FRT to ensure the quality of the FGD and School Environment Checklist processes at the start of REDI fieldwork. • GRM CET team leader undertakes periodic quality checks of the REDI team’s work, and the quantitative research methods specialist undertakes analysis of OVC program quantitative data and prepares report. October 22–25 • The entire Evaluation Study Team convenes in Surabaya, East Java, for in-depth data analy￾sis and identification of key findings and conclusions from the REDI FGD data. October 26–November 22 • GRM CET undertakes fieldwork in three provinces (DKI Jakarta, East Java, and NAD) and re￾turns to Jakarta for national-level meetings, data analysis, preparation for and conduct of the Stakeholders’ Meeting, and preparation of draft one of the evaluation report and final draft of the press release. Arriving at Findings, Conclusions and Recommendations • Following the completion of the REDI fieldwork and preliminary analysis of the FGD data, the Evaluation Study Team (including the GRM CET) will convene in Surabaya, East Java, for a three-day workshop to fully analyze the REDI FGD data. The team leader will lead this analyt￾ical process. The data analysis approach will consist of “eyeballing” and “pawing,” both con￾sidered valid qualitative data analysis techniques that are particularly relevant in situations where time is short. The meeting will be designed to enable full participation through incorpo￾rating discussions of findings as appropriate and will serve to validate the data collected and the preliminary analyses of the REDI team. A similar process will be used to analyze the data collected by the GRM CET. 68 | Page A matrix called “Identifying Findings, Key Findings, Conclusions and Recommendations” will be com￾piled by the GRM CET and the content in this matrix forms the basis of the content to be incorporated into the evaluation report. RIGOR TRIANGULATION A range of participants and methods and instruments are being utilized during this evaluation study. These will enable the EST to triangulate data. GOOD/BEST PRACTICE As described above, good and best practices in the conduct of the various qualitative methods will be utilized and the quality of these practices monitored. CONSTRAINTS AVAILABILITY OF PROGRAM DOCUMENTATION Program documentation is incomplete and time has been taken up collecting as much relevant docu￾mentation as possible. Much of the program documentation, particularly the training materials and products developed by stakeholders are only available in Bahasa Indonesia. Since two of the three GRM CET members do not read Bahasa Indonesia, this is a major issue. Translating all documents into English (e.g., using Google Translator) is not efficient or effective. The evaluation IEIS who is flu￾ent in Bahasa Indonesia will scan available documents and flag those that are the most important and discuss them with the team leader and key parts of relevant documents will be translated. Monitoring and evaluation documentation is scattered; time has been taken up investigating what monitoring and evaluation data exist and whether these data are useful for the purposes of the evalu￾ation study. The quantitative research methods specialist will review available monitoring and evalua￾tion documentation during the period between in-country fieldwork of the GRM CET and will produce a short paper discussing aspects of the documentation for inclusion in the evaluation report. ABSENCE OF BASELINE DATA The absence of baseline data, including data on the actual numbers of children with disabilities at a gross level and discrete data on children with specific types of disabilities, militates against the con￾duct of a survey. The evaluation study will attempt to mitigate this situation by looking into quantitative data on changes in numbers of children with special needs entering (and completing) primary educa￾tion from the start to the end of the program, if possible. The study will also mitigate this situation by gaining stakeholders’ perspectives on changes with respect to increased access and the quality of education provided to children with special needs in the program sites being visited during the field￾work. Baseline data on the capacity of institutions are also not available. However, a structured analytical tool to examine functional dimensions of special education departments in universities will be utilized. This tool is in development and will be available shortly. Assessment of capacity in a government di￾rectorate is more challenging. The team will rely on the qualitative data from interviews as well as as￾pects of the policy analytical framework tool to determine what type of change has occurred within MOEC and MORA. 69 | Page In an effort to limit recall bias and subjectivity inherent in an evaluation that is not able to draw on baseline data to document change, data collected will be triangulated and OVCP raw and processed quantitative data will be examined to the extent possible. AVAILABILITY OF OVC PROGRAM PERSONNEL As the OVCP closed down several months ago, the majority of the program’s personnel are no longer available. Although this situation is less than ideal, efforts will be made to connect with at least six of the field personnel who are based in and around Jakarta for a half-day meeting. The success of this meeting depends on the good will of the field personnel and their willingness to spend a weekend day with the GRM CET. It also depends on the availability of funds under the evaluation contract to pro￾vide these personnel with travel money (if necessary) and refreshments. APPROPRIATENESS AND AVAILABILITY OF EVALUATION STUDY PERSONNEL Further examination of the OVC program and the needs of USAID/Indonesia revealed that modifica￾tions were required in personnel originally proposed for the study and some increases required in the level of effort for some evaluation personnel. With delays in concluding a contract, evaluation person￾nel could not all be mobilized at the same time, resulting in a rather disjointed start to the evaluation exercise. Despite these early constraints, the evaluation study positions are now appropriate, LOE has been adjusted and reallocated in some instances and the REDI CET and FRT have been pre￾pared to undertake their fieldwork tasks. 70 | Page ANNEX 3: INSTRUMENTS ANNEX 3.1: FGD INSTRUMENTS ANNEX 3.1A: FGD INTRODUCTION AND WARM-UP GUIDE OVC Program Evaluation FGD Introduction and Warm-Up Guide Introduction and Warm-Up Questions Location of FGD (Village, Town, Subdistrict, District, Province) Name and ID Number of School Date of FGD Start Time & End Time Duration of FGD (in minutes) Number of Participants Number of Participants (F/M) with Disabilities (incl. Disability Type) Number of Female Participants Number of Male Participants Age Range of Respondents Name of Moderator Name of Note Taker Name of Quality Assurance Provider Introductory Remarks 71 | Page “My name is _________________________ and this is my colleague ___________________. We are carrying out an evaluation of a project funded by USAID called the Opportunities for Vulnerable Children Program, which was implemented by the Helen Keller Foundation. We work for an organiza￾tion called REDI (the Regional Economic Development), which is based in Surabaya. We are carrying out a study about the situations of women, men, girls, and boys in your community. We would like un￾derstand your situation. This information will become part of a large number of interviews with individ￾uals who have some knowledge of the Opportunities for Vulnerable Children Program in Indonesia. We would like to ask you a series of questions. We would like to hear about your specific experiences. We are asking about things that you have heard of or seen or know to be happening. The questions we are going to ask you today are about the ways in which children with disabilities are able to inter￾act in your community, school, and family. We ask you to please not use any names of people. You could say, “My friend is totally blind,” for ex￾ample. But please do not say, “Indra is totally blind.” Participation in the discussion is completely voluntary and you do not have to answer any questions that you do not want to answer. If you feel uncomfortable at any time during this discussion, it is fine to leave. If you are willing to stay, we would ask that you are also willing to share your views and opin￾ions with us. We will treat everything that you say today with respect, and your stories will be kept confidential. We will only share the answers you give as general answers based on those things that are shared with us by all of the people who speak to us. If we share a particular story, we will not present any infor￾mation that might identify you in anything that we write or record or photograph. We also ask you to keep the information shared during this discussion confidential. For example, if someone in the group shares a personal story, please respect that person’s privacy and do not tell others what was said here today. (Name) is taking notes while I lead the discussion. We take notes in order to make sure that we do not miss what you have to say. Is this all right with you? (Make sure everyone in the group shows their agreement clearly.) I would also like to record this discussion in case we miss something in our notes and need to check to make sure that we understood exactly what was being shared. Is this all right with you? (Make sure everyone in the group shows their agreement clearly.) We really want to hear what you have to say, and we want you to answer our questions in whatever way you want. There is no wrong answer to any question. We will make sure that everyone who wants to speak has a chance to speak. Sometimes, we may ask someone to let others speak. We expect our discussion to last for a maximum of two hours. Does anyone have any questions be￾fore we begin?” (Please note down any questions that people ask and your answers to the questions.) Warm-Up Questions We’d like to know some basic things about you all that will help us understand who you are. 1. How long has each of you lived in this village (or town)? 2. Are you all from the same village (or town)? (Note to moderator: If answer is “no,” get all names of other villages.) 3. Do any of you have any children? How many boys and how many girls? Do any of your boy or girl children have a disability? What is/are the disability/ies? 4. Are any of you married? Widowed? Single? Separated/Divorced? 72 | Page 5. What ethnic group do you belong to? 6. Do any of you practice a particular religion? 7. What grade level in school have you completed? 8. Can you read and write in Bahasa Indonesia? Do you speak English well? 9. Would you say that you are in very well-off situation, a comfortable situation, or a struggling situation? Thank you very much. Now, we would like to ask you some questions related to the Opportunities for Vulnerable Children Program and about the situation of people and children with disabilities in your families, your community, your schools, your district, and your province. We’d like you to think about any ways in which these things have changed for children with disabilities that you think of because of the Opportunities for Vulnerable Children Program. 73 | Page ANNEX 3.1B: FGD PROTOCOLS OVC Program Evaluation Interview Protocol: District-Level Education and Political Leaders (NOTE TO THE FGD RESEARCH TEAM: This set of questions will be asked of government person￾nel in all branches and levels of government who have knowledge of the education sector and the OVC Program. The questions will assess their knowledge of the OVC Program, the impact of the pro￾gram in their district and district’s schools, and how the district leadership may play a role in the pro￾gram’s sustainability. Probing questions are included to support the FGD researchers’ ability to obtain additional information related to the question.) Facilitator Lead-In Remarks “Now, we are going to ask you a series of questions about the OVC Program and your views about the program. Right now, we will be asking you a number of questions about changes that you have seen in yourself, in your schools in the district, in your communities, and in your district leadership. Can we begin?” Views on Situation in District 1. Please describe the situation in your district overall that existed before the OVC Program regarding (see bullets) _____________________. (NB: Please get the participants to think about different levels of the education system, that is, early childhood education, primary school, secondary school, voca￾tional/technical, university, and how children with disabilities are able to access and are accommodat￾ed at these various levels). • Access of children with disabilities to an education • Success of children with disabilities in gaining an education • Views of most people in the school and in the community about the education of children with disabilities Views on Self 2. Please think back to the time when you and your district first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for you in terms of your (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Has your attitude about children with disabilities changed? If yes, in what way? If no, why not? Have your beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about policies and programming for and teaching of children with disabili￾ties (Probing questions: Has your knowledge about policies and programming for children with disabilities improved? If yes, has your knowledge increased a huge amount, somewhat, or not at all? What things do you know now that you didn’t know before about policies and pro￾gramming for and teaching of children with disabilities?) • Knowledge about supporting other education administrators and district political lead￾ership in their efforts to support children with disabilities (Probing questions: Has your knowledge about how to support your colleagues in supporting children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you know now that you didn’t know before about supporting your col￾leagues in addressing the needs and capabilities of children with disabilities?) • Behaviors when supporting the needs and building the capabilities of children with disabilities 74 | Page (Probing question: If you have not changed your supporting behaviors, why not?) • Behaviors for supporting other education administrators and district leadership in their efforts to address the needs and capabilities of children with disabilities (Probing questions: Have your behaviors for supporting your colleagues in addressing the needs and capabilities of children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you do now that you didn’t do before to support your colleagues in addressing the needs of children with disabilities?) Views on District’s School Personnel 3. Please think back to the time when you and your district’s schools first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the pre￾sent. What are the most noticeable changes that have come about as a result of the program for your school personnel in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of your colleagues about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your colleagues about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities?) • Behaviors in general when dealing with children with disabilities (Probing questions: Have your colleagues’ behaviors toward children with disabilities changed? If yes, in what ways? If no, why not?) • Behaviors when teaching children with disabilities (Probing questions: If your colleagues have not changed their teaching behaviors, why do you think they have not?) • Behaviors of teachers who have not been involved in OVC Program support (Probing questions: What evidence have you seen that other teachers not involved with the program have adopted techniques that they may have heard about or seen OVC Program teachers or teacher trainers using?) • School leadership behaviors for addressing the education needs of children with disa￾bilities (Probing questions: Have your school leaders passed any decrees or policies relating to spe￾cial education? Relating to inclusive education? If yes, what are they? If not, why not? Are there any other ways your school administrators and school committee members are showing support for educating all children with disabilities? Views on Communities 4. Please think back to the time when you and your district first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your district’s communities overall that have come about as a re￾sult of the program in terms of (see bullets) _____________________ of most of the members of your district’s communities? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of community members about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of community members about children with disabili￾ties changed? If yes, would you say their knowledge increased a huge amount, somewhat, or 75 | Page not at all? What things do you think they know now that they didn’t know before the OVC Pro￾gram about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of people in your communities toward children with disabilities changed? If yes, in what ways? If no, why not?) Views on District Leadership 5. Please think back to the time when you and your district first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your district’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your district passed any regulations or policies relating to special ed￾ucation? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? Views on Province’s Leadership 6. Please think back to the time when you and your district first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your province’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your province passed any regulations or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 76 | Page Views on Interventions 7. When you think about the different parts of the OVC Program—work with universities and pre￾service teacher education, in-service teacher professional development, inter-school linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy devel￾opment, and community outreach, as well as other activities—what program activities do you think were the most successful and why? 8. The OVC Program closed several months ago. Which OVC Program activities are continuing through your school community’s leadership? Which activities are continuing through your district leaders’ efforts? Which activities are continuing through other organizations’ efforts (NB: What organi￾zations are these)? 9. When you think about the different parts of the OVC Program—for example, work with universities and pre-service teacher education, in-service teacher professional development, inter-school linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy development, and community outreach, as well as other activities—what program activities do you think were the most successful and why? 10. What expansion has taken place in activities started by the OVC Program in your schools overall? In your district’s communities? In your district? In your province? What new activities not part of the OVC Program have started up because people were inspired by the program? 11. Have you noticed any negative effects from activities that were started by the OVC Program in your schools overall? In your district’s communities? At the district level? Across the province? Why do you think these negative effects came about? 12. What particular activities of the OVC Program were low or no cost that you believe could be repli￾cated on a large scale? Why? Have you replicated any of these activities? Why or why not? Views on Situation for All Children with Disabilities 13. As a result of the OVC Program, has the education situation for all children with disabilities in your district improved a great deal, somewhat, not very much, or not at all? Has the education situation im￾proved for some types of children with disabilities but not for other types of children? If the situation has improved only for some types of children, which types of children have a better situation and which types of children do not? 14. As a result of the OVC Program, has the family situation for all children with disabilities in your dis￾trict improved a great deal, somewhat, not very much, or not at all? 15. Are children with disabilities at a greater risk of being victims of different forms of abuse, including sexual and gender-based violence? If yes, why do you think so? Are children with certain types of disabilities at greater risk of being victims of different forms of human rights abuse, including sexual and gender-based violence? If yes, why do you think so? Are girls with disabilities at greater risk than boys? Are girls with certain types of disabilities at greater risk than other girls? If yes, why do you think so? Advice to Decision Makers 16. If Helen Keller International staff were sitting with you today, what advice would you give to the HKI staff about how they could have made the program even better? (NB: It is very likely that every group is going to say “give more money,” so be prepared for this and ask a probing question that gets at what they think more money would help accomplish. Also, remind the group about the various components of the program—(1) preservice teacher education by improving the capacity of university special education departments; (2) in-service teacher professional development; (3) the school link￾ages approach, which consists of the special education school, model school, and satellite schools and inclusive education trainers and the resource teachers; (5) policy development; and (6) public outreach through media—and try to get them to give specific suggestions for improvements and changes to the components and the associated activities.) 77 | Page 17. If the Government of Indonesia (provincial and national levels) were to replicate the OVC Pro￾gram, what advice would you give the GOI to make the program and its impact even stronger? Views on OVC Program Impact 18. Overall, do you feel that the OVC Program did the most good at the level of the individual child, the family, the community, the school, the district government, the provincial government, or the na￾tional government? Why? (NB: Get the participants to provide concrete examples, not generalities.) 19. Are you familiar with any other models for strengthening the capacity of the education system to address children with disabilities? If you are familiar with any other model, how does the OVC Pro￾gram model compare with the other model? Why? Views on Effort of Indirect Beneficiaries 20. If you were to give yourself a mark for how well you embraced and moved forward the OVC Pro￾gram activities in which you were involved, would you give yourself an “A,” a “C,” or an “E”? Why do you give this mark? (NB: Note the similarities in response and look for the discrepant cases and ask those individuals why they gave a less typical mark. Then ask those individuals who gave more typical marks for their reasons. Carry out this process for each of the following questions.) If you were to give your school community a mark, would you give it an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your community and its leaders a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your district leadership a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your provincial leader￾ship a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your national leadership a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? 21. What is your biggest success story from your involvement with the OVC Program? 22. What is your biggest failure story from your involvement with the OVC Program? Views on Participation in the FGD Thank you for your responses to these questions. Now, we would like to ask you about your views on participating in this discussion. 21. Have you ever participated in a discussion like this before? When? Where? What for? 22. How do you feel about the discussion today? Did you learn anything new? 23. How do you feel we did as interviewers? Did we help you feel comfortable about sharing things with us? Can you give us any suggestions on making the process better? 24. As a result of our discussion today, will you do anything differently in the future? CLOSING These are all of our questions for now. Do you have anything you would like to add? Do you have any questions for us? As we told you in the beginning, our discussion today is meant to help us learn about the successes and challenges that arose from the implementation of the OVC Program. We hope that what we learned from our discussion today will help make the education situation for chil￾dren with disabilities even better. Please remember that you agreed to keep any private information that you heard in this discussion to yourself. 78 | Page OVC Program Evaluation Interview Protocol: Special Education Teachers/Trainers (NOTE TO THE FGD RESEARCH TEAM: This set of questions will be asked of government person￾nel in all branches and levels of government who have knowledge of the education sector and the OVC Program. The questions will assess their knowledge of the OVC Program, the impact of the pro￾gram in their school community, and how the school community leadership may play a role in the pro￾gram’s sustainability. Probing questions are included to support the FGD researchers’ ability to obtain additional information related to the question.) FGD FACILITATOR INTRODUCTORY REMARKS “Now, we are going to ask you a series of questions about the OVC Program and your views about the program. Right now, we will ask you a number of questions about changes that you have seen in yourself, in your school, in your community, and in your district. Can we begin?” 1. Please describe the situation in your school community before the OVC Program regarding: • Access of children with disabilities to an education • Success of children with disabilities in gaining an education • Views of most people in the school and in the community about the education of children with disabilities 2. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for you in terms of your (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Has your attitude about children with disabilities changed? If yes, in what way? If no, why not? Have your beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about teaching children with disabilities (Probing questions: Has your knowledge about teaching children with disabilities improved? If yes, has your knowledge increased a huge amount, somewhat, or not at all? What things do you know now that you didn’t know before about teaching children with disabilities?) • Knowledge about supporting other teachers in their efforts to teach children with disa￾bilities (Probing questions: Has your knowledge about how to support your colleagues in teaching children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you know now that you didn’t know before about supporting your col￾leagues in teaching children with disabilities?) • Behaviors when teaching children with disabilities (Probing questions: If you have not changed your teaching behaviors, why have you not?) • Behaviors for supporting other teachers in their efforts to teach children with disabili￾ties (Probing questions: Have your behaviors for supporting your colleagues in teaching children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you do now that you didn’t do before to support your colleagues in teaching children with disabilities?) 3. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for your school col￾leagues in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities 79 | Page (Probing questions: Have the attitudes of your colleagues about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your colleagues about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities?) • Behaviors in general when dealing with children with disabilities (Probing questions: Have your colleagues’ behaviors toward children with disabilities changed? If yes, in what ways? If no, why not?) • Behaviors when teaching children with disabilities (Probing questions: If your colleagues have not changed their teaching behaviors, why do you think they have not?) • Behaviors of teachers who have not been involved in the OVC Program support (Probing questions: What evidence have you seen of other teachers not involved with the program adopting techniques that they may have heard about or seen OVC Program teach￾ers or teacher trainers using?) 4. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your community that have come about as a result of the program in terms of the (see bullets) _____________________ of the members of your community: • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of your fellow community members about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your community’s members about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, some￾what, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of people in your community toward children with dis￾abilities changed? If yes, in what ways? If no, why not?) • School leadership behaviors for addressing the education needs of children with disa￾bilities (Probing questions: Has your school leadership passed any decrees or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways your school administrators and school committee members are showing sup￾port for educating all children with disabilities? 5. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your district’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? 80 | Page • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your district passed any regulations or policies relating to special ed￾ucation? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 6. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your province’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your province passed any regulations or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 7. When you think about the different parts of the OVC Program—work with universities and pre￾service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy devel￾opment, and community outreach as well as other activities—what program activities do you think were the most successful and why? 8. The OVC Program closed several months ago. Which OVC Program activities are continuing through your school community’s leadership? Which activities are continuing through your district leaders’ efforts? Which activities are continuing through other organizations’ efforts (NB: What organi￾zations are these)? 9. When you think about the different parts of the OVC Program—for example, work with universities and pre-service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy development and community outreach and media, as well as other activities—what program activities do you think were the least successful and why? 10. What expansion has there been of activities that were started by the OVC Program in your school and community? In your district? In your province? What new activities that were not part of the OVC Program have started up because people were inspired by the program? 11. Have you noticed any negative effects from activities that were started by the OVC Program in your school and community? In your district? In your province? Why do you think these negative ef￾fects came about? 81 | Page 12. What particular activities of the OVC Program were low or no cost that you believe could be repli￾cated on a large scale? Why? Have you replicated any of these activities? Why or why not? 13. As a result of the OVC Program, has the education situation for all children with disabilities in your community improved a great deal, somewhat, not very much, or not at all? Has the education situation improved for some types of children with disabilities but not for other types of children? If the situation has improved only for some types of children, which types of children have a better situation and which types of children do not? 14. As a result of the OVC Program, has the family situation for all children with disabilities improved a great deal, somewhat, not very much, or not at all? 15. Are children with disabilities at a greater risk of being victims of different forms of abuse, including sexual and gender-based violence? If yes, why do you think so? Are children with certain types of disabilities at greater risk of being victims of different forms of human rights abuse, including sexual and gender-based violence? If yes, why do you think so? Are girls with disabilities at greater risk than boys? Are girls with certain types of disabilities at greater risk than other girls? If yes, why do you think so? 16. If Helen Keller International staff were sitting with you today, what advice would you give to the HKI staff about how they could have made the program even better? (NB: It is very likely that every group is going to say, “Give more money,” so be prepared for this and ask a probing question that gets at what they think more money would help accomplish. Also, remind the group about the various components of the program—(1) preservice teacher education by improving the capacity of university special education departments; (2) in-service teacher professional development; (3) the school link￾ages approach, which consists of the special education school, model school, and satellite schools and inclusive education trainers and the resource teachers; (5) policy development; and (6) public outreach through media—and try to get them to give specific suggestions for improvements and changes to the components and the associated activities.) 17. If the Government of Indonesia were to replicate the OVC Program, what advice would you give the GOI to make the program and its impact even better? 18. Overall, do you feel that the OVC Program did the most good at the level of the individual child, the family, the community, the school, the district government level, the provincial government level or the national government level? Why? (NB: Get the participants to provide concrete examples, not generalities.) 19. Are you familiar with any other models for strengthening the capacity of the education system to address children with disabilities? If you are familiar with any other model, how does the OVC Pro￾gram model compare with the other model? Why? 20. If you were to give yourself a mark for how well you embraced and moved forward the OVC Pro￾gram activities in which you were involved, would you give yourself an “A,” a “C,” or an “F”? (NB: Note the similarities in response and look for the discrepant cases and ask those individuals why they gave a very different mark. Then ask those individuals who gave more typical marks for their reasons. Car￾ry out this process for each of the following questions.) If you were to give your school community a mark, would you give it an “A,” a “C,” or an “F”? If you were to give your community and its leaders a mark, would you give them an “A,” a “C,” or an “F”? If you were to give your district leadership a mark, would you give them an “A,” a “C,” or an “F”? If you were to give your provincial leadership a mark, would you give them an “A,” a “C,” or an “F”? If you were to give your national leadership a mark, would you give them an “A,” a “C,” or an “F”? Views on Participation in the FGD Thank you for your responses to these questions. Now, we would like to ask you about your views on participating in this discussion. 21. Have you ever participated in a discussion like this before? When? Where? What for? 82 | Page 22. How do you feel about the discussion today? Did you learn anything new? 23. How do you feel we did as interviewers? Did we help you feel comfortable about sharing things with us? Can you give us any suggestions for making the process better? 24. As a result of our discussion today, will you do anything differently in the future? CLOSING These are all of our questions for now. Do you have anything you would like to add? Do you have any questions for us? As we told you in the beginning, our discussion today is meant to help us learn about the successes and challenges that arose from the implementation of the OVC Program. We hope that what we learned from our discussion today will help make the education situation for chil￾dren with disabilities even better. Please remember that you agreed to keep any private information that you heard in this discussion to yourself. 83 | Page OVC Program Evaluation Interview Protocol: Parents/Caregivers of Children with Disabilities [NOTE TO THE FGD RESEARCH TEAM: This set of questions will be asked of parents, relatives, or caregivers of children with disabilities who have benefitted from the OVC Program. The questions will assess their knowledge of the OVC Program, the impact of the program on their child and on them and their families, school, and community, and how the parents may play a role in the program’s sus￾tainability. Probing questions are included to support FGD researchers’ ability to obtain additional in￾formation related to the question.) FGD FACILITATOR LEAD-IN REMARKS “Now, we are going to ask you a series of questions about the OVC Program and your views about the program. Right now, we will be asking you a number of questions about changes that you have seen in yourself, in your school, in your community, and in your district. Can we begin?” 1. Please describe the situation in your school community before the OVC Program regarding: • Access of children with disabilities to an education • Success of children with disabilities in gaining an education • Views of most people in the school and in the community about the education of children with disabilities 2. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for you in terms of your (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Has your attitude about children with disabilities changed? If yes, in what way? If no, why not? Have your beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about helping children with disabilities to succeed (Probing questions: Has your knowledge about helping children with disabilities improved? If yes, has your knowledge increased a huge amount, somewhat, or not at all? What things do you know now that you didn’t know before about helping children with disabilities to suc￾ceed?) • Knowledge about supporting other parents and families in their efforts to help their children with disabilities to succeed (Probing questions: Has your knowledge about how to support other families and children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you know now that you didn’t know before about supporting other parents and families in helping their children with disabilities to succeed?) • Behaviors when helping children with disabilities (Probing questions: If you have not changed your helping behaviors, why have you not?) • Behaviors for supporting other parents and caregivers in their efforts to help their children with disabilities to succeed (Probing questions: Have your behaviors for supporting other parents and caregivers of chil￾dren with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you do now that you didn’t do before to support other parents and caregivers of children with disabilities?) 3. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for school person￾nel in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities 84 | Page (Probing questions: Have the attitudes of your colleagues about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your colleagues about children with disabilities changed? If yes, would you say their knowledge in￾creased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities?) • Behaviors in general when dealing with children with disabilities (Probing questions: Have your colleagues’ behaviors toward children with disabilities changed? If yes, in what ways? If no, why not?) • Behaviors when teaching children with disabilities (Probing questions: If your colleagues have not changed their teaching behaviors, why do you think they have not?) • Behaviors of teachers who have not been involved in the OVC Program support? (Probing questions: What evidence have you seen of other teachers not involved with the program adopting techniques that they may have heard about or seen OVC Program teach￾ers or teacher trainers using?) 4. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your community that have come about as a result of the program in terms of the (see bullets) _____________________ of the members of your community: • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of your fellow community members about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your community’s members about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, some￾what, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of people in your community toward children with dis￾abilities changed? If yes, in what ways? If no, why not?) • School leadership behaviors for addressing the education needs of children with disa￾bilities (Probing questions: Has your school leadership passed any decrees or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways your school administrators and school committee members are showing sup￾port for educating all children with disabilities? 5. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your district’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? 85 | Page • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your district passed any regulations or policies relating to special ed￾ucation? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 6. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your province’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your province passed any regulations or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 7. When you think about the different parts of the OVC Program—work with universities and pre￾service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy devel￾opment, and community outreach as well as other activities—what program activities do you think were the most successful and why? 8. The OVC Program closed several months ago. Which OVC Program activities are continuing through parents’ advocacy and leadership? Which activities are continuing through your school’s leadership? Which activities are continuing through your community’s leadership? Which activities are continuing through your district leaders’ efforts? Which activities are continuing through other organi￾zations’ efforts (NB: What organizations are these)? 9. When you think about the different parts of the OVC Program—for example, work with universities and pre-service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy development and community outreach and media, as well as other activities—what program activities do you think were the least successful and why? 10. What expansion has there been of activities that were started by the OVC Program in your school? In your community? In your district? In your province? What new activities that were not part of the OVC Program have started up because people were inspired by the program? 11. Have you noticed any negative effects from activities that were started by the OVC Program in your school? In your community? In your district? In your province? Why do you think these negative effects came about? 86 | Page 12. What particular activities of the OVC Program were low or no cost that you believe could be repli￾cated on a large scale? Why? Have you replicated any of these activities? Why or why not? 13. As a result of the OVC Program, has the education situation for all children with disabilities in your community improved a great deal, somewhat, not very much, or not at all? Has the education situation improved for some types of children with disabilities but not for other types of children? If the situation has improved only for some types of children, which types of children have a better situation and which types of children do not? 14. As a result of the OVC Program, has the family situation for all children with disabilities improved a great deal, somewhat, not very much, or not at all? 15. Are children with disabilities at a greater risk of being victims of different forms of abuse, including sexual and gender-based violence? If yes, why do you think so? Are children with certain types of disabilities at greater risk of being victims of different forms of human rights abuse, including sexual and gender-based violence? If yes, why do you think so? Are girls with disabilities at greater risk than boys? Are girls with certain types of disabilities at greater risk than other girls? If yes, why do you think so? 16. If Helen Keller International staff were sitting with you today, what advice would you give to the HKI staff about how they could have made the program even better? (NB: It is very likely that every group is going to say, “Give more money,” so be prepared for this and ask a probing question that gets at what they think more money would help accomplish. Also, remind the group about the various components of the program—(1) preservice teacher education by improving the capacity of university special education departments; (2) in-service teacher professional development; (3) the school link￾ages approach, which consists of the special education school, model school, and satellite schools and inclusive education trainers and the resource teachers; (5) policy development; and (6) public outreach through media—and try to get them to give specific suggestions for improvements and changes to the components and the associated activities.) 17. If the Government of Indonesia were to replicate the OVC Program, what advice would you give the GOI to make the program and its impact even better? 18. Overall, do you feel that the OVC Program did the most good at the level of the individual child, the family, the community, the school, the district government level, the provincial government level or the national government level? Why? (NB: Get the participants to provide concrete examples, not generalities.) 19. Are you familiar with any other models for strengthening the capacity of the education system to address children with disabilities? If you are familiar with any other model, how does the OVC Pro￾gram model compare with the other model? Why? 20. If you were to give yourself a mark for how well you embraced and moved forward the OVC Pro￾gram activities in which you were involved, would you give yourself an “A,” a “C,” or an “E”? Why do you give this mark? (NB: Note the similarities in response and look for the discrepant cases and ask those individuals why they gave a very different mark. Then ask those individuals who gave more typ￾ical marks for their reasons. Carry out this process for each of the following questions.) If you were to give your school community a mark, would you give it an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your community and its leaders a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your district leadership a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your provincial leadership a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? If you were to give your national leadership a mark, would you give them an “A,” a “C,” or an “E”? Why do you give this mark? 21. As parents and caregivers, what are the three biggest changes you have seen in your child as a result of the efforts of the OVC Program? How important are these changes in your child? 87 | Page Views on Participation in the FGD Thank you for your responses to these questions. Now, we would like to ask you about your views on participating in this discussion. 21. Have you ever participated in a discussion like this before? When? Where? What for? 22. How do you feel about the discussion today? Did you learn anything new? 23. How do you feel we did as interviewers? Did we help you feel comfortable about sharing things with us? Can you give us any suggestions for making the process better? 24. As a result of our discussion today, will you do anything differently in the future? CLOSING These are all of our questions for now. Do you have anything you would like to add? Do you have any questions for us? As we told you in the beginning, our discussion today is meant to help us learn about the successes and challenges that arose from the implementation of the OVC Program. We hope that what we learned from our discussion today will help make the education situation for chil￾dren with disabilities even better. Please remember that you agreed to keep any private information that you heard in this discussion to yourself. 88 | Page OVC Program Evaluation Interview Protocol: Regular Education Teachers (NOTE TO THE FGD RESEARCH TEAM: This set of questions will be asked of government person￾nel in all branches and levels of government who have knowledge of the education sector and the OVC Program. The questions will assess their knowledge of the OVC Program, the impact of the pro￾gram in their school community, and how the school community leadership may play a role in the pro￾gram’s sustainability. Probing questions are included to support the FGD researchers’ ability to obtain additional information related to the question.) FGD FACILITATOR LEAD-IN REMARKS “Now, we are going to ask you a series of questions about the OVC Program and your views about the program. Right now, we will be asking you a number of questions about changes that you have seen in yourself, in your school, in your community, and in your district. Can we begin?” 1. Please describe the situation in your school community before the OVC Program regarding: • Access of children with disabilities to an education • Success of children with disabilities in gaining an education • Views of most people in the school and in the community about the education of children with disabilities 2. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for you in terms of your (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Has your attitude about children with disabilities changed? If yes, in what way? If no, why not? Have your beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about teaching children with disabilities (Probing questions: Has your knowledge about teaching children with disabilities improved? If yes, has your knowledge increased a huge amount, somewhat, or not at all? What things do you know now that you didn’t know before about teaching children with disabilities?) • Knowledge about supporting other teachers in their efforts to teach children with disa￾bilities (Probing questions: Has your knowledge about how to support your colleagues in teaching children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you know now that you didn’t know before about supporting your col￾leagues in teaching children with disabilities?) • Behaviors when teaching children with disabilities (Probing questions: If you have not changed your teaching behaviors, why have you not?) • Behaviors for supporting other teachers in their efforts to teach children with disabili￾ties (Probing questions: Have your behaviors for supporting your colleagues in teaching children with disabilities increased a huge amount, somewhat, or not at all? Why do you think so? What things do you do now that you didn’t do before to support your colleagues in teaching children with disabilities?) 3. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes that have come about as a result of the program for your school col￾leagues in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities 89 | Page (Probing questions: Have the attitudes of your colleagues about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your colleagues about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities?) • Behaviors in general when dealing with children with disabilities (Probing questions: Have your colleagues’ behaviors toward children with disabilities changed? If yes, in what ways? If no, why not?) • Behaviors when teaching children with disabilities (Probing questions: If your colleagues have not changed their teaching behaviors, why do you think they have not?) • Behaviors of teachers who have not been involved in the OVC Program support? (Probing questions: What evidence have you seen of other teachers not involved with the program adopting techniques that they may have heard about or seen OVC Program teach￾ers or teacher trainers using?) 4. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your community that have come about as a result of the program in terms of the (see bullets) _____________________ of the members of your community: • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of your fellow community members about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabilities changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of your community’s members about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, some￾what, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of people in your community toward children with dis￾abilities changed? If yes, in what ways? If no, why not?) • School leadership behaviors for addressing the education needs of children with disa￾bilities (Probing questions: Has your school leadership passed any decrees or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways your school administrators and school committee members are showing sup￾port for educating all children with disabilities? 5. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your district’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? 90 | Page • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your district passed any regulations or policies relating to special ed￾ucation? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 6. Please think back to the time when you and your school first began to be involved with the OVC Program. Now, think about the how special education has worked from that time to the present. What are the most noticeable changes in your province’s education and political leadership that have come about as a result of the program in terms of their (see bullets) _____________________? • Attitudes and beliefs about children and people with disabilities (Probing questions: Have the attitudes of district leaders about children with disabilities changed? If yes, in what way? If no, why not? Have their beliefs about children with disabili￾ties changed? If yes, in what way? If no, why not?) • Knowledge about children with disabilities (Probing questions: Has the knowledge of district leaders about children with disabilities changed? If yes, would you say their knowledge increased a huge amount, somewhat, or not at all? What things do you think they know now that they didn’t know before the OVC Program about children with disabilities? • Behaviors when dealing with children with disabilities (Probing questions: Have the behaviors of district leaders in your community toward children with disabilities changed? If yes, in what ways? If no, why not?) • Leadership behaviors for addressing the education needs of children with disabilities (Probing questions: Has your province passed any regulations or policies relating to special education? To inclusive education? If yes, what are they? If not, why not? Are there any other ways education administrators and district-level politicians are showing support for educating all children with disabilities? 7. When you think about the different parts of the OVC Program—work with universities and pre￾service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy devel￾opment, and community outreach as well as other activities—what program activities do you think were the most successful and why? 8. The OVC Program closed several months ago. Which OVC Program activities are continuing through your school community’s leadership? Which activities are continuing through your district leaders’ efforts? Which activities are continuing through other organizations’ efforts (NB: What organi￾zations are these)? 9. When you think about the different parts of the OVC Program—for example, work with universities and pre-service teacher education, in-service teacher professional development, interschool linkages among special education schools, model schools and satellite schools, resource centers/rooms, policy development and community outreach and media, as well as other activities—what program activities do you think were the least successful and why? 10. What expansion has there been of activities that were started by the OVC Program in your school and community? In your district? In your province? What new activities that were not part of the OVC Program have started up because people were inspired by the program? 11. Have you noticed any negative effects from activities that were started by the OVC Program in your school and community? In your district? In your province? Why do you think these negative ef￾fects came about? 91 | Page 12. What particular activities of the OVC Program were low or no cost that you believe could be repli￾cated on a large scale? Why? Have you replicated any of these activities? Why or why not? 13. As a result of the OVC Program, has the education situation for all children with disabilities in your community improved a great deal, somewhat, not very much, or not at all? Has the education situation improved for some types of children with disabilities but not for other types of children? If the situation has improved only for some types of children, which types of children have a better situation and which types of children do not? 14. As a result of the OVC Program, has the family situation for all children with disabilities improved a great deal, somewhat, not very much, or not at all? 15. Are children with disabilities at a greater risk of being victims of different forms of abuse, including sexual and gender-based violence? If yes, why do you think so? Are children with certain types of disabilities at greater risk of being victims of different forms of human rights abuse, including sexual and gender-based violence? If yes, why do you think so? Are girls with disabilities at greater risk than boys? Are girls with certain types of disabilities at greater risk than other girls? If yes, why do you think so? 16. If Helen Keller International staff were sitting with you today, what advice would you give to the HKI staff about how they could have made the program even better? (NB: It is very likely that every group is going to say, “Give more money,” so be prepared for this and ask a probing question that gets at what they think more money would help accomplish. Also, remind the group about the various components of the program—(1) preservice teacher education by improving the capacity of university special education departments; (2) in-service teacher professional development; (3) the school link￾ages approach, which consists of the special education school, model school, and satellite schools and inclusive education trainers and the resource teachers; (5) policy development; and (6) public outreach through media—and try to get them to give specific suggestions for improvements and changes to the components and the associated activities.) 17. If the Government of Indonesia were to replicate the OVC Program, what advice would you give the GOI to make the program and its impact even better? 18. Overall, do you feel that the OVC Program did the most good at the level of the individual child, the family, the community, the school, the district government level, the provincial government level or the national government level? Why? (NB: Get the participants to provide concrete examples, not generalities.) 19. Are you familiar with any other models for strengthening the capacity of the education system to address children with disabilities? If you are familiar with any other model, how does the OVC Pro￾gram model compare with the other model? Why? 20. If you were to give yourself a mark for how well you embraced and moved forward the OVC Pro￾gram activities in which you were involved, would you give yourself an “A,” a “C,” or an “E”? (NB: Note the similarities in response and look for the discrepant cases and ask those individuals why they gave a very different mark. Then ask those individuals who gave more typical marks for their reasons. Car￾ry out this process for each of the following questions.) If you were to give your school community a mark, would you give it an “A,” a “C,” or an “E”? If you were to give your community and its leaders a mark, would you give them an “A,” a “C,” or an “E”? If you were to give your district leadership a mark, would you give them an “A,” a “C,” or an “E”? If you were to give your provincial leadership a mark, would you give them an “A,” a “C,” or an “E”? If you were to give your national leadership a mark, would you give them an “A,” a “C,” or an “E”? Views on Participation in the FGD Thank you for your responses to these questions. Now, we would like to ask you about your views on participating in this discussion. 21. Have you ever participated in a discussion like this before? When? Where? What for? 92 | Page 22. How do you feel about the discussion today? Did you learn anything new? 23. How do you feel we did as interviewers? Did we help you feel comfortable about sharing things with us? Can you give us any suggestions for making the process better? 24. As a result of our discussion today, will you do anything differently in the future? CLOSING These are all of our questions for now. Do you have anything you would like to add? Do you have any questions for us? As we told you in the beginning, our discussion today is meant to help us learn about the successes and challenges that arose from the implementation of the OVC Program. We hope that what we learned from our discussion today will help make the education situation for chil￾dren with disabilities even better. Please remember that you agreed to keep any private information that you heard in this discussion to yourself. 93 | Page ANNEX 3.1C: FGD REPORT TEMPLATE INSTRUCTIONS 1. Immediately after the FGD, the facilitator and note taker should draft the FGD report. Part I: Summary Contextual and Biographical Data Table 1: FGD Descriptor Data Table FGD target group (circle) Regular Teachers (Guru Umum) Model School Teachers Resource Teachers Inclusive Education Trainers School Community Leaders District Leaders FGD location (village/town, district, province) FGD date FGD duration (start/end time and total minutes) Language(s) used by facilita￾tor in FGD Language(s) used by partici￾pants in FGD Facilitator name Note taker name 94 | Page Table 2: FGD Participant Summary Table FGC Participant Sex Age SES Disability type Village District Ethnicity Education level* Literate in Bahasa Indonesia (Y/N) Any English language proficiency (Y/N) Position (or type of work for parents and commu￾nity leaders) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. TOTAL 95 | Page * Education Level Descriptors: No formal education; primary-part completed; primary-completed; lower secondary-part completed; lower secondary-completed; upper secondary-part completed; upper secondary-completed; bachelor’s degree-part completed; bachelor’s degree-completed; master’s degree-part completed; master’s degree-completed; doctorate-part completed; doctorate-completed. 96 | Page Part II FGD Research Team Direct Observation of Surroundings and Participants INSTRUCTIONS: 1. Write down your insights into the information below. Doing so will help you remember details and interpret what was said. • Perspective on how the FGD was conducted (were you happy with your performance, did an￾ything unusual or notable happen before, during or after the FGD, etc.) • Impressions of the setting • Any changes to the question sequence or questions themselves and any additional interest￾ing questions that you incorporated into the FGD • Impressions of the participants, including any observations about indications of wealth, pov￾erty, education, enthusiasm and body language. Part III: FGD Patterns and Themes Summary INSTRUCTIONS 1. Make sure your FGD notes are complete and as close to verbatim as possible. Listen to the FGD audio recording to see if you have missed any information and fill in any missing information in the notes. 2. Read the FGD Protocol Questions Summary Table 3 below. Re-read your FGD notes and use a marking system to identify common patterns and themes and any “discrepant case”—a discrepant case is something that emerges that is very much outside the common experience expressed by most of the group. It is good to probe more on such cases because you often uncover very interesting in￾formation. 3. Identify and write down key findings in the categories listed below under the Key Findings section. Remember, key findings are clusters of findings that reveal a pattern or theme that emerge when a particular issue is being discussed. 4. Remember that we are trying to demonstrate change (positive or negative) or lack of change that came about as a result of the OVC Program (the before and after picture) and your FGD key findings should make it possible to see the change and the extent of change from participants’ perspectives. 97 | Page Table 3: FGD Protocol Questions Summary Table What ex￾isted be￾fore Changes be￾cause of OVC Pro￾gram Sustainability of changes Grade A, C, F Individual’s a) Attitudes and Be￾liefs, b) Knowledge, c) Behav￾iors and Practices Grade for you Colleagues’ a) attitudes and be￾liefs, b) knowledge, c) behaviors and practices Grade for col￾leagues School/community’s a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) pol￾icies Grade for school/community District leaders’ a) attitudes and beliefs, b) knowledge, c) behav￾iors and practices, d) policies Grade for district Provincial leaders’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) pol￾icies Grade for prov￾ince National leaders’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) pol￾icies Grade for MOEC Children with disabilities’ a) atti￾tudes and beliefs, b) knowledge, c) behaviors and practices NA Children without disabilities’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices NA Most successful activi￾ties/interventions NA NA NA Least successful activi￾ties/interventions NA NA NA Multiplier effects NA NA NA Sexual and gender-based vio￾lence against children with dis￾abilities NA NA NA Advice for HKI NA NA NA Advice for GOI NA NA NA OVC Program did the most good at the level of the individ￾ual child, family, community, school, district government, NA NA NA 98 | Page provincial government, or na￾tional government Other models NA 99 | Page KEY FINDINGS SUMMARY • Individual’s a) attitudes and beliefs, b) knowledge, c) behaviors and practices • Colleagues’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices • School/community’s a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) pol￾icies • District leaders’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) policies • Provincial leaders’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) poli￾cies • National leaders’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices, d) poli￾cies • Children with disabilities’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices • Children without disabilities’ a) attitudes and beliefs, b) knowledge, c) behaviors and practices • Most successful activities/interventions and common views on why • Least successful activities/interventions and common views on why • Multiplier effects and common views on why • Sexual and gender-based violence against children with disabilities and common views on why • Advice for HKI and common views on why • Advice for GOI and common views on why • OVC Program did the most good at the level of the individual child, family, community, school, district government, provincial government, or national government, and common views on why • Other models and common views on these compared with OVC Program 100 | Page Part IV: Your Emerging Conclusions INSTRUCTIONS Based on your analysis of your key findings from this particular FGD, write down the conclusions you draw about the following evaluation questions. If the particular evaluation question is not applicable to this FGD, simply write “NA” for “not applicable.” 1. Has the coordination of policy, planning, and funding between the national, provincial, and district levels improved? In what ways and for what reasons? In what ways have national, provincial, and district governments devoted additional efforts to promoting inclusive educa￾tion environments? What differences and similarities are there between provinces and dis￾tricts? Do stakeholders perceive any efforts as being significant, modest, or inconsequential? 2. Has the capacity of universities improved? In what ways and for what reasons? 3. Have the in-service training programs improved? In what ways and for what reasons? How has the program’s focus on the professional development of teachers to teach students with special needs changed classroom practices? Do stakeholders perceive any efforts as being significant, modest, or inconsequential? 4. Has the program resulted in a shift in the environment for children with special needs in pro￾gram communities? In what ways have attitudes about special needs children changed in communities through the efforts of community members (schools, parents, community lead￾ers) to promote equal access to social interactions for boys and girls with special needs within the community and beyond? Do stakeholders perceive any efforts as being significant, mod￾est, or inconsequential? In what ways have parents gotten involved in helping their special needs children gain access to inclusive schools? Do stakeholders perceive any efforts as be￾ing significant, modest, or inconsequential? 5. What differences and similarities in the treatment of inclusive education between secular and religious schools (madrasahs, in particular) and between OVC Program model schools and satellite schools? 6. In what ways did the OVC programs account for and address the gender-specific challenges and needs of girls and boys in their program? What changes came about as a result of the at￾tention to gender? Do stakeholders perceive any efforts as being significant, modest, or in￾consequential? 7. What aspects of the program are likely to be sustained and what aspects are not likely to be sustained and for what reasons? 101 | Page ANNEX 3.2: KEY PARTICIPANT INTERVIEW PROTOCOL Location of FGD (village, town, subdistrict, district, province) Name and ID number of school Date of interview Start time & end time Duration of interview (in minutes) Number of participants Number of participants (F/M) with disabilities (incl. disability type) Number of female participants Number of male participants Age range of respondents Names of interviewers Introduction and Warm-Up Questions “My name is _________________________ and this is my colleague ___________________. I am the Evaluation XX (position) and Y is the Evaluation ZZ. We are independent external evaluators and we are carrying out an evaluation of a program funded by USAID called the Opportunities for Vulner￾able Children Program, which was implemented by the Helen Keller Foundation from 2003 to 2013. We would like to ask you a series of questions. We would like to hear about your specific experiences. We are asking about things that you have heard of or seen or know to be happening. The questions we are going to be asking you today are about the ways in which children with disabilities are able to interact in your community, your school, and your family. This information will become part of a large number of interviews analyzed together of individuals who have some knowledge of the Opportunities for Vulnerable Children Program in Indonesia. We would like to start off with some broad areas of discussion and then, depending on what emerges from that discussion, we may follow up with some more specific questions. By the end of this interview, we’d like to have a sense of your overall observations for NS, SP, and Karachi on the sociocultural and economic landscape and how it has changed over past decades or so. We want to hear about the following: 1. Observations on changes in education, health, and livelihoods 2. Interesting strategies being used in these three sectors 102 | Page 3. Observations on changes in gender relations, roles, and norms 4. Observations on the nexus between radicalization and gender Privacy and Confidentiality Remarks We ask you to please not to use any names of people. You could say, “My friend is totally blind,” for example. But please do not say, “Indra is totally blind.” Participation in this interview is completely voluntary, and you do not have to answer any questions that you do not want to answer. If you feel uncomfortable at any time during this discussion, it is fine to leave. If you are willing to stay, we would ask that you are also willing to share your views and opin￾ions with us. We will treat everything that you say today with respect and your stories will be kept confidential. We will only share the answers you give as general answers based on those things that are shared with us by all of the people who speak to us. If we share a particular story, we will not present any infor￾mation that might identify you in anything that we write or record or photograph. We also ask you to keep the information shared during this discussion confidential. For example, if someone shares a personal story, please respect that person’s privacy and do not tell others what was said here today. We take notes in order to make sure that we do not miss what you have to say. Is this all right with you? (Make sure everyone in the group shows their agreement clearly.) I would also like to record this discussion in case we miss something in our notes and need to check to make sure that we understood exactly what was being shared. Is this all right with you? (Make sure everyone in the group shows their agreement clearly.) We really want to hear what you have to say, and we want you to answer our questions in whatever way you want. There is no wrong answer to any question. We will make sure that everyone who wants to speak has a chance to speak. Sometimes, we may ask someone to let others speak. We expect our discussion to last for a maximum of one hour. Does anyone have any questions before we begin?” (Please note down any questions that people ask and your answers to the questions.) Warm-Up Questions We’d like to know some basic things about you all that will help us understand who you are. 1. How long have you been in your current position and what did you do before this position? 2. Do any of you have any children? How many boys and how many girls? Do any of your boy or girl children have a disability? What is/are the disability/ies? Thank you very much. Now, we would like to ask you some questions related to the Opportunities for Vulnerable Children Program and about the situation of people and children with disabilities in your families, your community, your schools, your district, and your province. We’d like you to think about any ways in which these things have changed for children with disabilities that you think of because of the Opportunities for Vulnerable Children Program. Follow topical areas of FGD Instruments. 103 | Page ANNEX 3.3: SCHOOL ENVIRONMENT CHECKLIST LEMBAR OBSERVASI SEKOLAH PENYELENGGARA PENDIDIKAN INKLUSIF (INCLUSIVE SCHOOL OBSERVATION) A. Pelaksanaan Observasi/Observation Implementation 1. Tanggal pelaksanaan Date of observation : 2. Waktu Time : Mulai: Start Selesai : Finish 3. Durasi Pengamatan Duration: : ………………Jam (hour)………………. . menit (minutes) ……………... detik (seconds] 4. Nama pengamat Observer : 1. 2. B. Informasi Sekolah/School Information 1. Nama sekolah School name : 2. Nomor Induk Sekolah School register number : 3. Nomor SK sebagai SPPI (Apabila Ada) IE school decree (if there is any) : 1. 2. 3. 4. 5. 4. Yang menerbitkan SK IE school decree sources : 1. 2. 3. 4. 5. 5. Nama Kepala SPPI IE school principal name : 6. Alamat sekolah School address : Desa/Kota (village/town): Kecamatan (subdistrict): 7. Tel/fax/email number Phone/fax number : Tel: Fax: Email: 8. Latar belakang pendidi￾kan guru dan status kepegawaian Education background of the teacher and sta￾tus : DII Lk/M: Pr/F: PNS: Non-PNS: PNS: Non-PNS: S1- Non PLB (non-SNE) Lk/M: Pr/F: PNS: Non-PNS: PNS: Non-PNS: S-1 PLB/SNE Lk/M: Pr/F: PNS: Non-PNS: PNS: Non-PNS: S-2 and S3 Lk/M: PNS: Non-PNS: 104 | Page Pr/F: PNS: Non-PNS: Total Lk/M: Pr/F: PNS: Non PNS: 9. Guru Pembimbing Khu￾sus (GPK) Resource teachers : Ada (available) Tidak Ada (not available) Lk/M : ________ Orang • PNS: Non-PNS: • PLB: Non-PLB: • Guru SLB: • Guru Sekolah: • Lainnya: Pr/F: ________ Orang • PNS: Non-PNS: • PLB: Non-PLB: • Guru SLB: • Guru Sekolah: • Lainnya: 10. Tenaga kependidikan lain School education per￾sonnel : 1. 2. 3. 4. Lk ______ Pr _______ Lk ______ Pr _______ Lk ______ Pr _______ Lk ______ Pr _______ PNS ____ Non ____ PNS ____ Non ____ PNS ____ Non ____ PNS ____ Non ____ 11. Jumlah kelas Number of classrooms : ……………………………. ruang kelas classrooms Jumlah rombongan Belajar Number of group learn￾ers : ……………………………. Rombel 12. Jumlah siswa ABK Number children with disabilities : Lk/M : ___________ Pr/F : ___________ Total : 13. Jumlah siswa se￾luruhnya Number of students : Lk/M : ___________ Pr/F : ___________ Total : 14. Bagan struktur SPPI IE school structure : Tolong difoto bagan struktur SPPI Please take a picture of the IE school structure 105 | Page C. Informasi Sarana: Prasarana Sekolah / School Facilities Information Berdasarkan Pedoman Khusus Penyelenggaraan Pendidikan Inklusi DepDikNas 2007: Kebutuhan dan Pengelolaan Sarana dan Prasarana Pendidikan, Bab II, bagian B (Based on national IE specific guideline MOEC: School facilities management and needs) Amati secara umum, bagaimana sekolah mempersiapkan lingkungan yang aman dan nyaman bagi semua anak. Berilah tanda (√) pada kolom fasilitas yang tersedia. Observe generally how school sets up safeness and comfort environment for all children. Put check (√) into the blank block. LingkunganSekolah School environment Ketersediaan Availability Kondisi Condition Selokan / saluran air School water canal Ada Available Tidak Ada Not available Tertutup Closed Terbuka Opened Halaman School yard Ada Available Tidak Ada Not available Tertutup Closed Terbuka Opened Gedung sekolah School building Bertingkat Story Ya /Yes Tidak/ No Tangga ke lantai atas (jika sekolah bertingkat) Stairs to 2nd floor Tangga Stairs Lerengan Ramp Lebar/Width < 90 cm Lebar/Width > 90 cm Pintu kelas Classroom door Ada Available Tidak Ada Not available Lebar/Width < 90 cm Lebar/ Width > 90 cm Membuka ke luar Open outward Membuka ke dalam Open inward Berfungsi dengan baik Functions well Tidak berfungsi dengan baik Does not function well Pencahayaan di kelas Classroom illumination Sangat terang Very light Cukup terang Enough Kurang terang Less light Sirkulasi udara di kelas Classroom air circulation Baik Good Tidak baik Bad Bersumber dari? Source Jendela Window Lubang an￾gin Rooster Lainnnya: Others 106 | Page Jendela kelas Classroom window Ada Available Tidak Ada Not available Berfungsi dengan baik Functions well Tidak berfungsi dengan baik Does not function well Cara membu￾ka How opens Geser Sliding Dorong Push out￾ward Keatas Upward Kebawah Downward Gorden Curtain Ada Available Tidak Ada Not available Dapat dijangkau anak? Easier to be reached Ya Yes Tidak No Alat Pembelajaran di kelas Classroom learning facili￾ties Sesuai postur anak According to the size of child Ujung alat pembelajaran Classroom facilities corner Ya /Yes Tidak/ No Lancip Sharp Tumpul Rounded Ruang sumber Resource room Ada Available Tidak Ada Not available Ruang khusus Special room Ruang lain di￾fungsikan Other classroom to be functioned Bersih? Clean? Ya Yes Dapat dilewati kursi roda navigable by wheelchair (lebar ≥ 90 cm) Ya /Yes Tid￾akNo Tidak/No Lerengan Ramp Ada Available Tidak Ada Not available Lerengan menghubungkan: Ramps connecting to a. _________________ b. _________________ c. _________________ d. _________________ Pegangan Hand drill Ada Available Tidak Ada Not available WC Toilet Ada Available Tidak Ada Not available Mudah dilewati kursi roda navigable by wheelchair Ya Yes Tidak No Terbuka Opened Tertutup Closed Kondisi Condition Bersih Clean Kotor Dirty 107 | Page Kantin Canteen Ada Available Tidak Ada Not available Mudah dilewati kursi roda navigable by wheelchair Ya Yes Tidak No Kondisi Condition Bersih Clean Kotor Dirty UKS School health program fa￾cilities Ada Available Tidak Ada Not available Bersih Clean Kotor Dirty Perpustakaan Library Ada Available Tidak Ada Not available Mudah dilewati kursi roda navigable by wheelchair Ya Yes Tidak No Apakah ada tempat ber￾main dengan sarana untuk pengembangan keterampi￾lan gerak yang tersedia (contoh: ayunan, trampo￾line, dll)? Is there any playground with “gross motor” equip￾ment (swing, trampoline, etc.)? Ada Available Tidak Ada Not available Catatan Note Apakah ada jadwal pelaja￾ran yang terlihat? Is there a classroom schedule visible? Ada Available Tidak Ada Not available Catatan Note Apakah ada jadwal individ￾ual yang terlihat? Are there any individual schedules available? Ada Available Tidak Ada Not available Catatan Note Apakah ruang kelas diatur dengan baik? Is the classroom well or￾ganized? Ada Available Tidak Ada Not available Catatan Note Apakah ada ruang baca? Is there a reading corner? Ada Available Tidak Ada Not available Catatan Note 108 | Page Apakah ada ruangan kecil untuk membantu konsen￾trasi siswa pada suatu ak￾tivitas tertentu? Do you have cubicles for concentrating on specific activities? Ada Available Tidak Ada Not available Catatan Note Apakah ada papan penilaian perilaku siswa? Do you have a behavior chart? Ada Available Tidak Ada Not available Catatan Note Apakah materi pembelaja￾ran diletakkan pada tempat yang mudah dijangkau oleh anak-anak? Are learning materials placed in locations that are easy for children to reach? Ya Yes Tidak No Catatan Note Apakah ruang serta materi pembelajaran menggunakan warna yang berbeda dengan kontras yang jelas? Are different and highly contrasting colors used in the resource center and learning materials? Ya Yes Tidak No Catatan Note Apakah ruang dan materi pembelajaran menggunakan tekstur dan bervariasi? Are a variety of textures used in the resource center and materials? Ya Yes Tidak No Catatan Note Apakah ada mainan untuk merangsang keterampilan motorik halus (puzzle, Legos, balok-balok? Are there toys to stimulate fine motor skills (puzzles, Legos, blocks)? Ya Yes Tidak No Catatan Note Apakah ada riglet dan pena yang tersedia? Is there a slate and stylus available? Ya Yes Tidak No Catatan Note 109 | Page Apakah ada tongkat yang tersedia? Is there a cane available? Ya Yes Tidak No Catatan Note Apakah ada CCTV? Is there a CCTV? Ya Yes Tidak No Catatan Note Apakah ada computer da￾lam ruang kelas? Is there a computer in the classroom? Ya Yes Tidak No Catatan Note Apakah ruang kelas aman untuk semua siswa? Is the classroom safe for each student? Ya Yes Tidak No Catatan Note 110 | Page ANNEX 3.4: POLICY ANALYSIS INSTRUMENTS ANNEX 3.4A: POLICY ANALYSIS MATRIX Process Paper Price Tags Selected for DKI Jakarta East Jakarta South East Java Nganjuk Tuban Aceh Lhokseuma Pidie There is Division for Special and People Programs National - Jakarta 111 | Page ANNEX 3.4B: POLICY TRACKING TABLE FOR PROVINCIAL AND DISTRICT LEVELS ON INCLUSIVE EDUCATION FOR CHILDREN WITH DISABILITIES / SPECIAL NEEDS P1: Country: P2: APR reporting year: P3: Select one or more of the following policy areas (check all that apply)  Access for children with disabilities/special needs  Special teachers and/or resource teachers’ provision  Teachers’ capacity development  Student’s enrolment which include children with disabilities  Gender sensitive program  Reducing violence towards children including children with disabilities  Strategic Information  Financing  Other (please describe): ____________________________________. P4: Policy problem (e. g., parents and children with disabilities lack timely ac￾cess to quality education for all through inclusive education): P5: Related service delivery indicators: P6: Who or what body is lead￾ing/managing this policy reform pro￾cess? P7: Who or what body is responsible for monitoring this policy reform? 112 | Page P8: List and describe N-OVC contribu￾tions to this policy reform process in the past year: P9: Policy reform progress Provide a response for each policy stage and answer the associated narrative questions for each stage marked as “complete” or “in progress”: Stage 1 Identify baseline policy issue(s)/ problem(s) Stage 2 Develop policy in￾tervention & docu￾ment Stage 3 Official govern￾ment endorsement of policy Stage 4 Implement policy Stage 5 Evaluation of pol￾icy impact on health  Complete  In progress  Not begun  Complete  In pro￾gress  Not begun  Complete  In progress  Not begun  Complete  In progress  Not begun  Complete  In progress  Not begun STAGE In the stages marked below, answer all of the questions as “complete” or “in pro￾gress.” RESPONSE Stage 1: Identify baseline policy is￾sue(s)/problem(s) 1(a): Describe specific policy issue(s) /problem(s) needing reform (e.g., parents and children with disabilities lack timely access to quality education for all through inclusive education). 1(b): Has a formal analysis been conduct￾ed of this problem? If so, identify the document and its author(s). 1(c): List and describe significant stake￾holder meetings, consultations, briefings, or similar events held regarding this poli￾cy in the past 12 months. 113 | Page 1(d): Additional optional narrative update for Stage 1. Stage 2: Develop policy interven￾tion /document 2(a): Describe the policy intervention(s) chosen to address the problem(s) identi￾fied in 1(a) above. 2(b): What national standards informed the development of your policy interven￾tion(s)? Describe how these standards are or are not incorporated into the policy intervention documents. 2(c): What is the formal mechanism for each policy intervention described in 2(a) (e. g., budget, act of parliament, cabinet policy, guideline)? What policymaker or policy body will adopt or enact the above policy intervention(s)? 2(e): What barriers, if any, remain to adopt or enact the policy? 114 | Page 2(f): Are revisions to other laws or poli￾cies required to ensure policy harmoniza￾tion? If so, what are they? 2(g): Additional optional narrative update for Stage 2. Stage 3: Official government en￾dorsement of pol￾icy 3(a): Have the policy intervention(s) de￾scribed in 2(a) been officially adopt￾ed/enacted? Which, if any, of the interven￾tion(s) described in 2(a) have not been adopted? 3(b): List the titles of policy documents adopted/enacted, the date on which they were adopted, and the name of the person or body that adopted the policy docu￾ment. 3(c): Additional optional narrative update for Stage 3. Stage 4: Imple￾ment policy 4(a): Has a costed implementation plan been developed and adopted? If not, de￾scribe current status and next steps. 4(b): Have resources been allocated to implement the policy intervention(s)? If not, describe current status and next steps. 115 | Page 4(c): Has the policy and/or implementa￾tion aids to implement the policy been disseminated to those charged with im￾plementing it? 4(d): Describe how this policy is being adopted and implemented at the district or other local levels. 4(e): Describe major implementation ac￾tivities conducted in the past 12 months. 4(f): Additional optional narrative update for Stage 4. Stage 5: Evalua￾tion of policy im￾plementation 5(a): Describe monitoring activities con￾ducted in the past year (e. g. , monitoring committee meetings held, surveys con￾ducted, progress reports produced) 5(b): Have any barriers to implementation or monitoring implementation been identi￾fied? If yes, how do you plan to address them? 5(c): Describe any plans to evaluate the impact of this policy on health outcomes and the results of any completed evalua￾tions. 116 | Page 5(d): Additional optional narrative update for Stage 5. 117 | Page ANNEX 4: FINDINGS/CONCLUSIONS/RECOMMENDATIONS ANALYTICAL GRID Questions Findings Key Findings Conclusions Recommendations 1. Has the coordination of policy, planning, and funding among the national, provin￾cial, and district levels improved? a. In what ways and for what reasons? Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? b. What differences and similarities exist between provinces and districts? c. What differences and similarities exist between provinces and districts? d. In what ways have national, provincial, and district governments devoted addi￾tional efforts to promoting inclusive ed￾ucation environments? 2. Has the capacity of universities improved? 118 | Page a. In what ways and for what reasons? 3. Have the in-service training programs im￾proved? a. In what ways and for what reasons? b. How has the OVC Program’s focus on the professional development of teach￾ers to teach students with special needs changed classroom practices? c. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? 4. Has the program resulted in a shift in the environment for children with special needs in program communities? a. In what ways have attitudes about spe￾cial needs children changed in com￾munities through the efforts of commu￾nity members (schools, parents, com￾munity leaders) to promote equal ac￾cess to social interactions for boys and girls with special needs within the community and beyond? 119 | Page b. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? c. In what ways have parents gotten in￾volved in helping their special needs children gain access to inclusive schools? d. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? 5. What differences and similarities exist in the treatment of inclusive education be￾tween secular and religious schools (mad￾rasahs, in particular) and between OVC Program model schools and satellite schools? 6. In what ways did the OVC programs ac￾count for and address the gender-specific challenges and needs of girls and boys in their program? a. What changes came about as a result of the attention to gender? b. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? 120 | Page 7. What aspects of the program are likely to be sustained and not sustained, and for what reasons? 8. Has the coordination of policy, planning, and funding between the national, provin￾cial, and district levels improved? a. In what ways and for what reasons? In what ways have national, provincial, and district governments devoted addi￾tional efforts to promoting inclusive ed￾ucation environments? b. What differences and similarities exist between provinces and districts? c. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? d. Has the capacity of universities im￾proved? In what ways and for what reasons? 9. Have the in-service training programs im￾proved? 121 | Page a. In what ways and for what reasons? 10. How has the program’s focus on the pro￾fessional development of teachers to teach students with special needs changed classroom practices? a. Do stakeholders perceive any efforts as being significant, modest, or incon￾sequential? 122 | Page ANNEX 5: OVC PROGRAM PHASES Figure 1 below illustrates the phases and the key intervention foci in and/or carried through the OVC program. Figure 1: OVC Program Phases