July 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by SSG Advisors. It was authored by Brenda Sinclair, Jennie Campos and Marissa Kimsey. FINAL PERFORMANCE EVALUATION OF USAID/DOMINICAN REPUBLIC’S EFFECTIVE SCHOOLS PROGRAM (ESP) This publication was produced for review by the United States Agency for International Development by SSG Advisors, through USAID Contract # AID-OAA-M-14-00024. This report was prepared by: SSG Advisors 1 Mill Street, Suite 201 Burlington, VT 05401 SSG Advisors Contact: Olya Jerard Senior Operations and Project Manager Tel: (802) 735-1169 Email: olya@ssg-advisors.com FINAL PERFORMANCE EVALUATION OF USAID/DOMINICAN REPUBLIC’S EFFECTIVE SCHOOLS PROGRAM (ESP): FINAL EVALUATION REPORT July 2015 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS List of Tables .......................................................................................................................... iii Acronyms and Abbreviations ...............................................................................................iv Executive Summary.................................................................................................................v Evaluation Purpose and Evaluation Questions....................................................................................................... v Project Background ..................................................................................................................................................... v Evaluation Questions, Design, Methods and Limitations.................................................................................... v Findings, Conclusions and Recommendations...................................................................................................... vi Purpose and Evaluation Questions.......................................................................................1 Evaluation Purpose....................................................................................................................................................... 1 Evaluation Questions................................................................................................................................................... 1 Project Background.................................................................................................................2 Results Framework ...................................................................................................................................................... 3 Evaluation Methods and Limitations ...................................................................................3 Evaluation Design ......................................................................................................................................................... 3 Methodology.................................................................................................................................................................. 4 Participant Selection .................................................................................................................................................... 4 Sampling Methodology ................................................................................................................................................ 5 Data Collection............................................................................................................................................................. 5 Data Analysis................................................................................................................................................................. 6 Limitations...................................................................................................................................................................... 6 Findings, Conclusions and Recommendations....................................................................7 Evaluation Question 1: Achievement of Results................................................................................................... 7 Evaluation Question 2: Reading Results................................................................................................................10 Evaluation Question 3: Safe Schools Results .......................................................................................................15 Evaluation Question 4: Special Needs Results.....................................................................................................19 Evaluation Question 5: Effectiveness .....................................................................................................................23 Evaluation Question 6: Sustainability: Contributions to DR Education System ..........................................27 Final Thoughts and Conclusions......................................................................................... 30 Annex A. Evaluation Statement of Work ........................................................................ 31 I. Objective of the Evaluation ..................................................................................................................................31 II. Background – Effective Schools Program (ESP)..............................................................................................31 III. Evaluation Focus and Key Questions...............................................................................................................33 IV. General Evaluation Methodology .....................................................................................................................33 V. Team Requirements .............................................................................................................................................34 VI. Deliverable.............................................................................................................................................................35 VII. Performance Requirements and Standards...................................................................................................37 VIII. Supporting Information.....................................................................................................................................38 IX. Illustrative Timeline (Assumes Monday to Friday as Billable Workdays)...............................................39 X. Key References and Reading for Evaluators...................................................................................................40 Annex B. Evaluation Planning Matrix ............................................................................... 42 Reading .........................................................................................................................................................................42 Safe Schools.................................................................................................................................................................43 Inclusive Education.....................................................................................................................................................44 Implementation Effectiveness ..................................................................................................................................45 ii Annex C. Evaluation Methods and Limitations............................................................... 46 Sampling Methodology ..............................................................................................................................................47 Annex D. Data Collection Instruments ............................................................................ 48 USAID Stakeholder Questionnaire........................................................................................................................49 National Stakeholder Questionnaire .....................................................................................................................51 District Stakeholder Questionnaire.......................................................................................................................55 School Principal Questionnaire...............................................................................................................................60 Spanish Language Teacher Questionnaire ............................................................................................................64 Spanish Language Classroom Observation Tool.................................................................................................68 Safe Schools School Staff Questionnaire...............................................................................................................74 Inclusive Education School Staff Questionnaire ..................................................................................................78 Parents Focus Group Tool.......................................................................................................................................83 Youth Focus Group Tool .........................................................................................................................................85 Annex E. Sources of Information....................................................................................... 87 Reference List .............................................................................................................................................................87 Key Informants............................................................................................................................................................90 Sites Visited..................................................................................................................................................................90 Annex F. Disclosure of Conflict of Interest Forms ......................................................... 91 Team Leader – Brenda Sinclair...............................................................................................................................91 Basic Education Specialist – Juanita Campos........................................................................................................92 Program Analyst – Marissa Kimsey ........................................................................................................................93 Local Specialist – Aida Hernández..........................................................................................................................94 List of Tables iii LIST OF TABLES Table 1. Evaluation Topics .....................................................................................................2 Table 2. Respondents by Type and Sample Size................................................................4 Table 3. School Selection Criteria ........................................................................................5 Table 4. Regions Sampled.......................................................................................................5 Table 5. Five-Year Results for Spanish, Math and Education Management .................7 Table 6. Reading Performance of Third and Fourth Grade Students ......................... 11 Table 7. School Selection Criteria ..................................................................................... 47 Table 8. Reading Comprehension Scores ......................................................................... 47 Table 9. Respondent Summary .......................................................................................... 90 Table 10. Schools Visited by Region.................................................................................. 90 Table 11. Types of Schools Visited .................................................................................... 90 Acronyms and Abbreviations iv ACRONYMS AND ABBREVIATIONS AMCHAM American Chamber of Commerce CAD Centro de Recursos para la Atención a la Diversidad CETT Centers for Excellence in Teacher Training COP Chief of Party DR Dominican Republic EGMA Early Grade Math Assessment EGRA Early Grade Reading Assessment ESP Effective Schools Program FCR Findings, Conclusions, and Recommendations GBV Gender-Based Violence IDB Inter-American Development Bank IE Inclusive Education IR Intermediate Result ISFODOSU Instituto Superior de Formación Docente Salome Urena LEER Lighting Excitement for Excellence in Reading M&E Monitoring and Evaluation MINERD Ministry of Education of the Dominican Republic MOE Ministry of Education PMP Performance Monitoring Plan PTA Parent-Teacher Association PUCMM Pontificia Universidad Católica Madre y Maestra RFP Request for Proposals SERCE Second Regional Comparative and Explanatory Study SOW Statement of Work SKII Semi-Structured Key Informant Interviews SL Spanish Language SS Safe Schools TEF Teachers and Local Governance Project TERCE Third Regional Comparative and Explanatory Study UNESCO United Nations Educational, Scientific and Cultural Organization USAID United States Agency for International Development Executive Summary v EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The dual objectives of the Effective Schools Program (ESP) final performance evaluation are to assess the effectiveness of the ESP in achieving its performance goals and to inform the implementation of current and future USAID/Dominican Republic (DR) projects. The evaluation places particular emphasis on the Reading, Safe Schools (SS) and Inclusive Education (IE) program components, which are key activities in the upcoming USAID Lighting Excitement for Excellence in Reading (LEER) project. Towards these aims, the evaluation report has been organized to respond to six key evaluation questions: 1. Achievement of Results: Did the project meet its expected results and targets as described in the cooperative agreement? 2. Reading Results: How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted communities? What factors are causing reading scores to remain low overall, even in cases where progress was made under ESP? 3. Safe Schools Results: How did the Safe Schools activity perform for boys and girls in terms of improving safety and creating an environment of tolerance in targeted schools? Were conditions improved for learning as a result of Safe School activities? Was there any noticeable correlation between a safe learning environment and academic improvement, as measured through improvements in reading? 4. Special Needs Results: How did the ESP perform in terms of meeting the requirements of targeted Special Needs boys and girls? Were changes in enrollment or academic performance observed as a result of ESP activities with special needs children? 5. Effectiveness: How well was the project implemented, coordinated and managed? What were the challenges and lessons learned? 6. Sustainability: What are the perceived contributions to the DR public primary school system? PROJECT BACKGROUND The ESP is a $17-million program (2009-2015) funded by the United States Agency for International Development (USAID) and implemented through a cooperative agreement with the Pontificia Universidad Católica Madre y Maestra (PUCMM), working in collaboration with the Ministry of Education of the Dominican Republic (MINERD). The goal of ESP as stated in the cooperative agreement is “to significantly contribute to the improvement of quality learning of students in the first cycle of primary education” (USAID, 2010, p. 11). To achieve this goal, the program executed activities through five components: (1) education management, (2) in-service teacher training in Spanish and Mathematics, (3) monitoring and evaluation, (4) safe schools, and (5) inclusive education opportunities for children with special needs. The first three components were part of the original program design and were implemented in 400 schools. The IE and SS components were added in 2013 and piloted in 55 and 90 schools, respectively. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The evaluation design is a utilization-focused evaluation intended to provide recommendations and lessons learned to inform USAID/DR’s upcoming LEER project. The team employed a mixed-methods approach to data collection. Data were gathered through document review, semi-structured key informant interviews, focus groups with parents and youth, and classroom observations. A total of 506 respondents and 53 schools in six regions participated in the evaluation. The schools were selected using a purposeful sampling methodology. The criteria for selection included schools with all five ESP components, schools with high/low reading scores, schools with single-component SS or IE programming, and extended day schools. Inherent limitations in the evaluation reflect the program design such as overlapping components, predecessor projects and differing timeframes for Executive Summary vi implementation, which created challenges in attributing results to specific interventions. Including a mix of single-component and five-component schools assisted with identifying factors that contribute to results. Another limitation was sampling bias. There are a greater percentage of school-level respondents than district and national MINERD representatives due to the extent of involvement in the program and availability of participants during the evaluation period. Furthermore, due to the non￾experimental design, the evaluation findings are limited to ESP schools and districts sampled. Program data and staff were consulted to provide a holistic perspective of results achieved against targets. The findings are limited to the data available at the time of the evaluation and shared with the evaluation team. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation Q1: Did the project meet its expected results and targets as described in the cooperative agreement? Q1 Findings. The program was expected to train 5,500 teachers and improve 150,000 students’ performance in Mathematics and Spanish Language in 400 schools. It was also designed to improve management skills of 400 school principals and 150 district technicians as well as to educate parent groups on their roles and functions. ESP achieved these targets under Intermediate Result 1 (IR1) New Education Management Structure Strengthened, IR2 Literacy Learning Improved, and IR3 Math Learning Improved. In the three result areas, the program exceeded targets for training beneficiaries and for improving the competencies of teachers and students in Math and the Spanish Language. An analysis of ESP student achievement results found that students in third and fourth grade with at least one year in the ESP program performed better than their peers in the control group. For instance, fourth graders with two years in ESP improved in reading comprehension by 10.8 percentage points in 2014. The difference between the improvement observed in the comparison group (6.5%) and ESP (10.8%) was 4.4% in 2014 and was statistically significant. There were no specific targets set for the SS component. However, ESP conducted gender-based violence (GBV) awareness trainings in 90 schools and tracked 42 cases of violence in 25 schools. Eighty percent of teachers in Santiago and approximately 60% of teachers in Santo Domingo reported a reduction in violence on a post-intervention survey. The IE component targeted to diagnose and treat 500 students in 100 schools. The program properly diagnosed 272 students as having special needs in 55 schools and treated 50 students through provision of two surgeries and 48 pairs of eyeglasses. Short-term interventions were implemented due to time and resource constraints. With regards to the M&E component, the program assessed teachers’ and students’ competencies in reading and math each year of the program. Results of student achievement tests were disseminated to the Vice Minister of Quality Assurance and Evaluation Director, district and regional directors, and principals. One-third of teachers received the student achievement data; however, it arrived after students had been promoted. The majority of teachers and principals did not receive the results of individual student performance. Teacher assessments from classroom observations, however, were disseminated to school principals and teachers through a signed copy of the observation feedback and debriefing meetings. Q1 Conclusions. The program achieved its five-year targets for expected results as outlined in the cooperative agreement. Student achievement results indicate that ESP met its objective of improving student performance in the Spanish Language and Mathematics. The SS intervention achieved an important first step by raising consciousness of GBV and reducing incidences of violence in schools. The IE component identified a large percentage of students with learning disabilities that required long-term interventions. Thus, the ESP was unable to fulfill its target of 500 for treatment and support. In terms of supporting evidence-based decision-making, student achievement assessment results were generally disseminated at all levels of MINERD. The data disseminated to schools reflected district-wide results, and were not disaggregated by student to inform teacher instruction or by school to support school Executive Summary vii decision-making. Q1 Recommendations. 1. Increase the timeframe for implementation of the IE and SS components; set realistic targets 2. Incorporate school performance data into school leadership trainings and student achievement results into teacher trainings in order to directly improve school performance. 3. Disseminate teacher and student results of performance evaluations to the district and national MINERD (e.g., Basic Education department for first cycle of primary education). Evaluation Q2: How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted communities? Q2 Findings. ESP has demonstrated tangible results in improving students’ reading achievement. An analysis of reading achievement data on third and fourth graders indicates that while fourth graders in the comparison group improved by 6.5 percentage points on average in reading comprehension, the fourth graders with two years in ESP improved by 8.9, 9.6, and 10.8 percentage points in 2011, 2012, and 2014 respectively. On average, ESP students performed 3.3% higher than the comparison group; the percentage increased each year of the program. While schools do not have baseline data, teachers and principals perceive an average increase of 65% in percentage of students able to read fluently with comprehension since the inception of the program. Consistent with ESP results, third grade achieved the highest gains on formative assessments, according to teachers. Student performance and motivation were found to decrease in fourth grade due to three factors: an influx of transfer students, insufficient materials aligned with the curriculum, and infrequent coaching compared to teachers in the lower grades. The factors most associated with high reading achievement ranked in order of frequency of responses were: (1) use of diverse teaching materials aligned with the curriculum, (2) teachers well￾trained in the ESP methodology, (3) application of an effective reading methodology, and (4) parental reinforcement of reading. On the other hand, nearly all respondents (teachers, principals and MINERD informants) stated that lack of parental involvement in reinforcing the reading methodology is the number one factor contributing to low reading achievement. Other factors included family problems and learning difficulties. Interviews with reading teachers revealed that 70% of teachers provide individualized attention, but only 30% differentiate tasks for students with special needs. A key threat to sustainability of the reading progress achieved under ESP is reverting to the old methodology when trained teachers transfer or when students advance to fifth grade. Q2 Conclusions. The ESP made significant strides in improving reading comprehension levels in the early grades through provision of well-trained teachers, an effective reading methodology, and distribution of diverse teaching and learning materials. Results are evident in classroom observations and test results. The results of classroom observations and interviews indicate that additional materials (workbook, teacher-guides and grade-appropriate library books) are needed, especially in fourth grade. The most significant challenges contributing to low achievement are that parents lack understanding of the new methodology and the majority of teachers are unable to independently develop differentiated tasks for special needs students. Finally, the evaluation team found that teacher transfers and discontinuity in the second cycle hamper the sustainability of ESP’s achievements. Executive Summary viii Q2 Recommendations. 1. Continue providing support in the ESP schools through building the capacity of Teacher Coordinators1 to mentor new and existing teachers. 2. Provide Teacher Guides, textbooks, grade-appropriate library books, and diagnostic/formative assessment booklets for fourth grade aligned with the curriculum. 3. Train parents in the reading methodology and in how to support their children's reading at home and in school; engage more parents in reading fairs and school-wide events through offering incentives and convenient schedules. 4. Provide more tools, games and manipulatives to attend to students with special needs. 5. Extend the program to the second cycle (to at least sixth grade) and sustain innovation circles to promote transfer of knowledge among new and well-trained teachers. 6. Provide compensation or special recognition to model teachers who replicate the program in their school or zone. Evaluation Q3: How did the Safe Schools activity perform for boys and girls in terms of improving safety and creating an environment of tolerance in targeted schools? Q3 Findings. As a result of SS interventions, respondents noted the following improvements in the school environment: (1) a reduction in bullying and fighting, (2) the development of codes of conduct in schools, and (3) improved school management. The most significant change observed by 60% of teachers and principals interviewed was better-disciplined students. Youth who have participated in SS activities were more aware of conflict resolution strategies and tended to be more respectful towards teachers and students. However, the evaluation team witnessed eight incidents of school violence in low-income, high crime areas. Use of abusive language is also still a problem according to 82% of students who participated in focus groups. Despite the challenges in many schools visited, there were numerous instances of exemplary schools. Some of the more notable positive changes occurred in schools where principals and teachers were strategic in their efforts to establish codes of conduct aligned with a “culture of peace” that focused on promoting values and respect in the learning environment. Changes were also achieved when teachers who monitor recess separated lower and upper primary students and involved youth and teachers in leading productive activities during recess. Additionally, schools with sustained participation of all stakeholders in the program activities (e.g., leadership training, reading certification training, or parent schools) exhibited the greatest improvements. While 44% of school respondents reported an improvement in academic performance as a result of SS interventions, the achievement was not solely attributable to SS activities, but rather to improved school leadership and the effective reading program. The SS component was credited with contributing to positive attitudes, improved concentration, and behavior in the classroom. Q3 Conclusions. There is evidence of improved sensitization of GBV and less bullying, but violence persists in schools in low-income, high crime areas. School environments that tolerate harmful, disruptive and unruly behavior pose serious threats to students’ learning and safety. The broader 'machismo' culture and abuse in the home and community influence student behavior and language. In order to create a safe school, there needs to be transformational change. School-wide norms and regulations involving all actors (e.g., principals, parents, teachers, youth) were found to have a noticeable 1 PUCMM formed “teacher coordinators” who are a teacher in a school elected for their leadership. They assume the role of “mentor” to provide training and coordination for lower primary grades. The teacher coordinators were developed in each school in collaboration with the ministry. Executive Summary ix impact on creating a safe learning environment. Long-term training in leadership, parenting, and classroom management are essential for creating a school-wide culture that promotes peace and respect. While the SS component cannot be directly correlated with reading achievement, it did contribute to improved classroom conduct. Q3 Recommendations: 1. Provide continuous SS training for all primary grade teachers, principals and parents. 2. Collaborate with MINERD and other relevant ministries (e.g., Ministry of Women) to define SS interventions that address GBV, sexual abuse, and other forms of violence based on the socio￾cultural context of the DR. 3. Address violence and bullying on a national scale. 4. Provide training for school personnel to understand the connection between violence and abuse, its impact on student achievement and how it is evidenced in undisciplined behavior. 5. Share best practices from model ESP schools that successfully changed school culture and climate to promote peace, responsibility, leadership, and gender equity. 6. Provide advanced training options for school principals and teachers; implement parent education programs in schools with low reading achievement and/or high rates of violence. 7. Tap the rich resources of schools around the globe that have successfully addressed school bullying. Provide models and book collections that depict “safe school behavior, attitudes, and skills” as well as strategies to cope, manage, and avert such behavior. Evaluation Q4: How did the ESP perform in terms of meeting the requirements of targeted Special Needs boys and girls? Q4 Findings. Working with the diagnostic instrument and pre-populated list of children with disabilities provided by the ministry, ESP conducted diagnostic evaluations of 553 children in 55 schools. Of those assessed, 272 were classified as having special needs. The majority of the cases were intellectual learning disabilities, which required long-term interventions. As a result of time and resource constraints, ESP treated 18% of cases through provision of short-term medical treatments. In the 24 schools visited, school staff informed that 115 children were diagnosed and 43% of the cases treated, representing a higher percentage than reported by ESP. The types of support provided included referrals to a specialist, provision of IE materials and tutoring. Intellectual and physical disabilities were not addressed due to limited human and physical resources. Over 90% of school staff participating in IE activities believe the program’s activities and duration were insufficient to address all the needs identified for students with disabilities. In general, schools do not have functioning IE classrooms, designated IE Coordinators, or sufficient knowledge and resources to detect and respond to learning disabilities. The majority of school staff trained claimed that the training was too short and general to support them in effectively exercising their role. About one-quarter of schools reported increased enrollment of children with special needs as a result of parents’ learning about the IE program or due to outreach from motivated teachers. However, 20% of parents interviewed did not believe in the school’s capability to provide for their children’s special needs. While 70% of schools reported improvement in overall academic achievement of students with special needs, and 40% in reading specifically, these results are likely attributed to differentiated instruction training, materials and coaching received under the ESP Reading component and prior CETT interventions. In schools with an IE Coordinator, classroom and materials, there was insufficient time to achieve real learning gains in reading. In order to improve learning of students with special needs, schools stressed the need for greater parental involvement, additional training, appointment of an IE Coordinator, and more follow-up assistance. Q4 Conclusions. There was insufficient time to provide treatment for all children diagnosed with learning or physical disabilities due to short-term interventions and an exhaustive diagnostic assessment. The lack of a dedicated IE Coordinator is severely hindering the schools’ ability to utilize all resources Executive Summary x available to address the special needs of all children. Training was insufficient to equip school staff and parents with the skills needed to detect, diagnose and treat children with disabilities or learning difficulties. The majority of schools are not well equipped to attract out-of-school children with disabilities. Although 40% of schools reported an improvement in reading, the results are attributed to CETT/ESP reading activities and not IE activities specifically. Schools require additional training, follow-up assistance, and parental support in order to improve the learning of children with special needs. Q4 Recommendations. 1. Conduct annual diagnostic assessments and develop individualized treatment plans in collaboration with medical professionals in the community. 2. Rather than aiming to make all schools inclusive, develop model IE schools within zones equipped with all necessary resources. 3. Provide IE materials and follow-up support for treatment of cases; continue building the capacity of teachers to engage parents, school staff, and community members in developing and implementing treatment plans. 4. Offer master degrees and specialized certificates in response to the demand for qualified IE Coordinators and teachers 5. Create support groups for teachers and parents to exchange experiences and successful strategies. 6. Replicate the CETT/ESP differentiated instruction model with other teachers and schools. Evaluation Q5: How well was the project implemented, coordinated and managed? What were the challenges and lessons learned? Q5 Findings. Fifty percent of key informants lauded PUCMM for their excellent management, training, efficiency and sustained support. The majority of beneficiaries (80%) rated the training workshops as excellent or very good. Teachers and principals were also deeply impressed with the coaching quality and consistency. It was evident from school visits and interviews that past experience on the CETT program, long-standing relationships with teachers, USAID and MINERD, and the utilization of existing models accelerated program results. PUCMM was further recognized for its strong financial management and cost-effective strategies. For instance, PUCMM leveraged public and private sector funds for expanding the Math, School Management and IE components. The key challenge experienced by the project, according to principals, was teachers’ initial resistance to change and trainings on Saturdays. School management experts claim it takes two to three years to create cultural and mental shifts. Secondly, teacher transfers pose threats to continuity of the reading methodology. The third most significant challenge cited by respondents was lack of coordination with MINERD in planning and implementation due to high turnover and shifting government priorities. In terms of sustaining the reading methods, teachers reported that the development of teaching aids is time-consuming; and multiple shifts prevent teachers from developing reading corners and print rich classrooms, which are essential for promoting reading achievement. With regards to the SS component, although ESP investigated cases of child abuse during the program, there was no formal requirement or mechanism for reporting cases of child abuse in most schools visited. Schools further lack partnerships with the community and government to monitor and address GBV. The key challenges the IE component faced were the delayed appointment of an IE Coordinator, lack of materials and inadequate follow-up support. Lessons learned from coordination efforts indicated that overlapping projects affect fidelity to implementation, minimize impact due to competing priorities, and complicate the reporting process. Q5 Conclusions. PUCMM had excellent technical and financial management and coordination of activities across the components. The majority of beneficiaries were very pleased with the intensive and effective support received from PUCMM. Although it takes two to three years to create a cultural shift, now that the program has achieved results, it has the potential for rapid scale-up in current ESP schools. Executive Summary xi Throughout the project, there was a lack of coordination with MINERD; yet the recent adoption of ESP methodology and coordination with PUCMM for scale-up shows promise for future collaboration. Additional resources and infrastructure are necessary to achieve fidelity to implementation in overcrowded and low-resourced schools. In many ESP schools, there was a lack of school-community partnerships and mechanisms to address violence and cases of child abuse. The appointment of IE Coordinators/special education teachers is critical for implementing the IE strategy. Q5 Recommendations. 1. Continue supporting the integration of ESP into MINERD to sustain and replicate the program’s success. 2. Extend the program in ESP schools with high turnover; offer training during the convenient times and locations to increase participation; and, build the capacity of trained teachers and district technicians to provide ongoing mentoring. 3. Work with MINERD to sustain trained teachers in the lower grades. 4. Develop a coordinated strategy for USAID, MINERD, and the implementing partner to collaborate on project design, implementation and evaluation. 5. Involve the MOE in the planning the chronogram of activities to avoid schedule conflicts and in order to communicate a unified message to schools. 6. Provide pre-printed stories per grade, manipulatives and laminated cards or other materials that can be recycled to reduce classroom preparation time; another cost-effective option is to incorporate materials development into training workshops. 7. Dedicate a classroom for safekeeping of reading materials and full implementation of the program (e.g., reading corners, classroom displays). 8. Establish school-community networks for systematic reporting of child abuse and GBV cases; work with the government to strengthen case management and law enforcement. 9. Work with MINERD to appoint and train IE Coordinators who can support students with special needs and assist teachers with differentiating activities in the classroom. 10. One implementer should be in charge of all interventions in a school to eliminate conflict and maintain fidelity to implementation. Evaluation Q6: What are the perceived contributions to the DR public primary school system? Q6 Findings. The most noteworthy achievement of ESP, per MINERD key informants, was the development of Math and Spanish curricula in the first cycle of primary education. As a result of ESP support, the Ministry is now replicating math and reading materials and trainings in non-ESP schools in the northern region. The most significant accomplishment according to school-level respondents was improved student achievement in Reading and Math. Classroom libraries, publishing of children’s books and reading competitions greatly contributed to the reading outcomes. ESP also refined the CETT professional development model and developed a cadre of specialized teachers, coaches and dynamic classrooms for replication. Another sustainable contribution of the program was the creation of specialization programs in IE, Reading and Leadership. Additionally, over 2,000 principals benefited from leadership training conducted with MINERD modules. Through the IE and SS components, ESP improved students’ self esteem, behavior, and communication skills, which contributed to greater acceptance of children with special needs and a reduction in bullying. As a result of IE and reading interventions, teachers have increased awareness of how to support students with special needs, but still require additional assistance with diagnosis, treatment and differentiating tasks in the classroom. Q6 Conclusions. The ESP contributed to strengthening reading and math curriculum design, instruction and student achievement. PUCMM is working with MINERD to integrate and scale-up the Executive Summary xii program, but more support is needed with the transition. There was insufficient time and scale for the SS and IE components to create systemic change, but there are signs of improvement. Q6 Recommendations. 1. Build the capacity of MINERD technical staff to fully integrate and sustain ESP-MINERD (e.g., train technicians in the classroom observation tools and methodology so they can continue to support teachers; promote retention of trained teachers in lower grades; continue training teacher coordinators to mentor new and existing teachers; institutionalize teacher innovation circles). 2. Develop a mechanism for sustaining the reading competitions and book publishing. 3. Replicate the program; provide schools and teacher-training institutes with videos of training sessions and dynamic classrooms along with training manuals so they can replicate the approach with new teachers. 4. Work closely with ISFODOSU (Instituto Superior de Formación Docente Salome Urena) to continue providing specialized degrees and with INAFOCAM to fund scholarships. 5. Integrate all components from the inception of the project under one results framework. Purpose and Evaluation Questions 1 PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE There are two key objectives of the Effective Schools Program (ESP) final performance evaluation. One is to assess the effectiveness of the Effective School Program in achieving its performance goals and its contributions to the Dominican Republic (DR) public primary education system. The second objective is to inform the implementation of ongoing USAID/DR work and upcoming projects (e.g., LEER), which support primary education and early grade reading achievement. The evaluation places particular emphasis on the Reading, Safe Schools (SS) and Inclusive Education (IE) program components, which are key activities in the upcoming USAID LEER project. The evaluation further aimed to determine the effectiveness of the SS and IE components, which were not assessed during the midterm evaluation. The final performance evaluation is aligned with the specific objectives of USAID’s Evaluation Policy for performance evaluation. As such, the evaluation aims to assess whether expected results were achieved at the project’s conclusion; how well the project was implemented, coordinated, and managed; and how well it was perceived and valued in terms of its contributions to the DR public primary school system. The findings will inform USAID and implementating stakeholders of effective strategies, best practices and lessons learned from the life of the project and will provide suggested approaches for sustaining and/or replicating ESP successes. The evaluation data collection took place in six regions of the Dominican Republic from April 26 to May 27, 2015. 2 EVALUATION QUESTIONS The final performance evaluation questions were designed based on USAID objectives outlined in the Statement of Work (SOW) in Annex A and USAID’s Evaluation Policy. The six principal evaluation questions are: 1. Achievement of Results: Did the project meet its expected results and targets as described in the cooperative agreement? 2. Reading Results: How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted communities? What factors are causing reading scores to remain low overall, even in cases where progress was made under ESP? 3. Safe Schools Results: How did the Safe Schools activity perform for boys and girls in terms of improving safety and creating an environment of tolerance in targeted schools? Were conditions improved for learning as a result of Safe School activities? Was there any noticeable correlation between a safe learning environment and academic improvement, as measured through improvements in reading? 4. Special Needs Results: How did the ESP perform in terms of meeting the requirements of targeted Special Needs boys and girls? Were changes in enrollment or academic performance observed as a result of ESP activities with special needs children? 5. Effectiveness: How well was the project implemented, coordinated and managed? What were the challenges and lessons learned? 6. Sustainability: What are the perceived contributions to the DR public primary school system? 2 As per the Evaluation Conflict of Interest Forms in Annex F, all members of the SSG evaluation team met the conditions of compliance. The forms indicated no conflict of interest with businesses or relationships with USAID technical staff that would influence the oversight of project implementation or the process of procurement of goods or services for USAID. Project Background 2 The first three questions (#1-3) focus on key issues intended to inform current and future implementation of USAID/DR programs, particularly related to improving reading skills, creating safe schools and addressing the special needs of children with disabilities. The last three questions (#4-6) seek to assess the effectiveness of the Effective Schools Program in achieving its performance goals and its contributions to the DR public primary education system. The evaluation SOW and USAID highlighted a number of other priorities for the final performance evaluation. Therefore, in addition to the six evaluation questions, the evaluation sought to address the following points. Table 1. Evaluation Topics Topic Where Addressed Reading pedagogical approaches: To better understand the debate on reading pedagogical approaches (whole word vs. phonics), which approach has the program adopted and why? Has the project tried a balanced approach? Question 2: Reading Results Recommendations on how to improve the effectiveness of both Safe Schools and Inclusive Education programming for shaping USAID/DR education support and other programs. Question 3: Safe Schools Results Question 4: Special Needs Results Question 5: Effectiveness Midterm evaluation recommendation #6: Ensure student and teacher evaluations are implemented, analyzed and results are disseminated to all levels of the education system to inform decision-making. Question 1: Achievement of Results Coordination with other USAID projects: • Are there any obstacles to coordination where there are multiple USAID projects (e.g., AMCHAM’s Basic Education Program) operating in the same area? What are the advantages of multiple USAID projects operating in the same geographic area? Does USAID achieve greater results when projects operate in the same area? • Has the ESP contributed to reducing citizen’s perception of insecurity in targeted areas, in coordination with other USAID/DR programming focused on youth, crime, prevention and institutional strengthening of the criminal justice system? Question 5: Effectiveness. The coordination and impact question is addressed in Q5. Citizens’ perception of insecurity could not be addressed directly since the ESP had limited involvement with the community and did not work directly with Alerta Joven or any other USAID project addressing crime reduction. Parental and youth perceptions of violence in the school (which is an extension of the community) and changes observed are discussed in Question 4. PROJECT BACKGROUND The ESP is a $17-million program (2009-20153) funded by the United States Agency for International Development (USAID) and implemented through a cooperative agreement with the Pontificia Universidad Católica Madre y Maestra (PUCMM), working in collaboration with the Ministry of Education of the Dominican Republic (MINERD). The ESP builds on the success of two past USAID/PUCMM projects: the Centers for Excellence in Teacher Training (CETT), which focused on literacy in grades 1-3, and the Activities of Teachers and Local Governance project (TEF), which focused 3 The ESP was extended past its original end date of September 2014 to April 2015. Evaluation Methods and Limitations 3 on math in grades 1-4. The goal of ESP was to improve student performance in reading and math in the first cycle of primary education, grades 1-4. The program worked through five components to achieve results: education management, in-service teacher training in Spanish and Mathematics, monitoring and evaluation, safe schools interventions and inclusive education opportunities for children with special needs. RESULTS FRAMEWORK According to the Results Framework, the ESP has six intermediate results (IRs): IR1.New education management structure strengthened; IR2.Literacy learning improved; IR3.Math learning improved; IR4.Learning evaluations capacity building and systems strengthening; IR5.Gender violence awareness created; and IR6.Strengthen technical capacity to education children with special needs. The five program components and six intermediate results directly contributed to achieving the USAID education strategic objective, which is to improve the quality of student learning in the Dominican Republic’s primary education system. The program ultimately contributes to USAID Assistance Objective 3, Increase Educational Objectives. The results framework and cooperative agreement did not include a development hypothesis at the Program Objective level providing overall direction and linking the sub-elements since the program components were not introduced simultaneously. The first four results were part of the original design and have five-year strategic results. Per the cooperative agreement, the program was expected to train 5,500 teachers to increase effectiveness in the first cycle of public primary education so that 150,000 students’ performance would increase in Mathematics and Spanish Language in 400 schools. The program was also designed to improve management skills of 400 school directors and 150 school technicians along with strengthening the participation of community leaders, civil society and parents in public education. In 2013, the Safe Schools (SS) and Inclusive Education (IE) components were added. The SS activity was piloted in 90 schools with the objective of providing practical and cognitive tools to reduce gender￾based violence (GBV) within schools and create safer learning environments. The IE component was introduced in 55 ESP schools to build the capacity of public schools to support school-age children with disabilities. The program worked closely with MINERD to achieve program milestones in order to result in better informed decisions for the creation of education policies. EVALUATION METHODS AND LIMITATIONS EVALUATION DESIGN The evaluation design is a utilization-focused evaluation. The evaluation questions and methods were specifically selected to obtain recommendations and lessons learned to inform USAID/DR’s upcoming LEER project, which will replicate Reading, Safe Schools and Inclusive Education components from the ESP. The evaluation also was designed to assess the performance of the ESP with a focus on Safe Schools and Inclusive Education components, which were not evaluated during the midterm evaluation. Finally, the evaluation relied on existing data, such as student achievement data and the midterm evaluation Evaluation Methods and Limitations 4 findings, in order to measure the overall performance of the program in a comprehensive and cost￾effective manner. METHODOLOGY The evaluation team employed mixed methods to inform each evaluation question within the timeline and budget parameters of the contract. An evaluation-planning matrix was prepared linking each evaluation question with data collection methods and sources of data (see Annex B: Evaluation Planning Matrix). Applying document review, semi-structured key informant interviews (SKII), focus groups and classroom observations, the evaluation methods yielded both quantitative and qualitative data about the performance of the ESP. The team conducted 21 key informant interviews with national and district stakeholders, 47 classroom observations (30 minutes each on average), 51 principal interviews, 49 Spanish Language teacher interviews, 24 Safe Schools teacher interviews, 27 Inclusive Education coordinator/teacher interviews, 18 participatory focus groups with 147 youth from the Safe Schools component, and 26 focus groups with 165 parents from ESP schools. See Annex E for list of key informants and schools visited. The process for instrument development and data collection methods are described in greater detail in Annex C. PARTICIPANT SELECTION In total, 506 respondents participated in the evaluation. The participants were selected with inputs from USAID and PUCMM. The midterm evaluation contact list served as a point of reference; however, the majority of the key informants were no longer tied to the project. As a result, there was a significant under-representation of MINERD staff (see Table 2). There was a fairly equal distribution of school representatives (teachers and principals), parents, and youth; each comprised one-third of respondents. However, given the limited extent of parental and youth involvement in the program, the evaluation findings largely reflect teachers’ and principals’ perspectives. Table 2 provides a breakdown of participants by type and sample size. Table 2. Respondents by Type and Sample Size Respondent Type Total % USAID 10 2% MINERD 9 1.8% Principals 51 10% SL teachers 49 10% SS teachers/administrators 24 5% IE teachers/administrators 27 5% Parents 165 33% Youth in grades 5-8 171 34% Total Respondents 506 100% Source: Fieldwork data. It is important to note that participants in the SS and IE components varied. For the SS component, trainees consisted of primary school teachers in grades 1-8, school counselors, teacher coordinators, and principals or deputy principals. The participants represented in the IE component included teachers, IE coordinators, counselors, psychologists and principals or deputy principals. Trainees under the IE component depended upon the workshop objectives; thus, those interviewed may not have been involved in all IE activities. Evaluation Methods and Limitations 5 SAMPLING METHODOLOGY The schools were selected using a purposeful sampling methodology. The criteria for selection included schools with all five ESP components, schools with high and low reading achievement, and schools that received only the SS or IE interventions. Schools ranking in the bottom 12 or in the top 10 on ESP reading comprehension tests administered to 3rd and 4th grade students in 2012 and 2014 were sampled. The team also visited six extended day schools. Table 3 shows the selection criteria and percentage of ESP schools represented in the sample. See Annex C for a description of the methodology used to select schools and cut-off scores for high and low reading achievement. Table 3. School Selection Criteria Criteria Evaluation Sample ESP School Population % of ESP Schools Evaluated 5-Component School 11 15 73% High-Reading Achievement* 14 12 117% Low-Reading Achievement* 15 10 150% Total Reading 49 400 12% Safe Schools 25 91 27% Inclusive Education 22 55 40% Extended Day School 6 25 24% Total Schools 53 400 13% Source: Fieldwork data. Six regions and 53 schools comprised the total evaluation sample, as shown in Table 4. Table 4. Regions Sampled Region Schools Visited Jarabacoa 12 La Vega 11 Moca 1 Puerto Plata 1 Santiago 19 Santo Domingo 9 Total Schools 53 Source: Fieldwork data. DATA COLLECTION Data collection took place from April 28 to May 22, 2015 in six regions and 53 schools. During the first week in country, the evaluation team conducted interviews with key informants at the national level, including USAID staff, PUCMM program implementers, and MINERD stakeholders involved in the program. At the end of the first week, the evaluation team carried out a pilot school visit in La Vega to ensure instrument reliability and consistent protocols. The instruments were subsequently finalized. During the three weeks of fieldwork, each team visited two schools per day. Teams visited schools in Jarabacoa, La Vega and Puerto Plata the first week; in Santiago and La Vega the second week; and in Santiago, La Vega and Santo Domingo the third week. The team collected data in 53 schools. Depending on school type, the teams conducted in-depth interviews with school principals and teachers trained in the Spanish Language, Safe Schools, and/or Inclusive Education components. In nearly every school visited, an ESP-trained teacher in grades 1-4 was observed and interviewed. Where possible, the evaluators held focus group discussions with parents involved in the ESP activities and facilitated a participatory youth exercise with students in grades 5-8 involved in the SS activity. Evaluation Methods and Limitations 6 The teams also met with District and Regional Directors in Jarabacoa, La Vega and Santiago, and with mentors in Santiago and Santo Domingo. DATA ANALYSIS Data from all sources were recorded and analyzed independently. The data from key informant interviews were consolidated by themes and triangulated to identify common responses among stakeholder groups. The data were then sorted to determine frequencies and percentages. The last step of data analysis was the development of a Findings, Conclusions and Recommendations (FCR) matrix aligned with the six evaluation questions. LIMITATIONS As with most performance evaluations, the non-experimental evaluation design limits the extent of generalizability of the findings. The evaluation data therefore reflect ESP schools and districts sampled. To provide a holistic perspective of the overall program results achieved, the evaluation team incorporated inputs from program coordinators, referenced ESP monitoring and evaluation data, and reviewed student achievement data (see Questions 1 and 2). The findings are limited to the data available at the time of the evaluation and shared with the evaluation team. The program design limited the extent to which causal attributions could be drawn to the ESP. For instance, the differing time periods for component interventions, prior CETT interventions, and the varied components in each school posed challenges to attributing results to specific ESP activities. To increase the reliability of the findings, a purposeful, stratified sampling approach was applied. The team visited schools with one component and those with all five components to determine whether multiple components produced greater impact than single interventions and to assess the true impact of the SS and IE components. Including a mix of single-component and five-component schools assisted with data analysis and drawing conclusions about which factors contributed to results. A further limitation was selection bias. As teachers and principals were more readily available and concentrated, there is a higher representation of these stakeholders. Parents’ lack of participation in the project components limited their inputs into the key questions. With regards to MINERD participation, MINERD staff were difficult to access due to high turnover, pressing schedules, and limited involvement in the project. Findings indicate that closer communication and coordination with national and district MINERD staff are recommended. In some cases, it was also difficult to verify activities that took place under the SS and IE components. For instance, participants trained under the IE component varied according to the workshop objectives; thus, those interviewed may not have been involved in all IE activities. Teacher transfers further complicated the evaluation. The evaluation teams frequently encountered instances in which the trained SS or IE person had been transferred, and so the person interviewed may not have been well informed of past training workshops. Due to frequent teacher transfers, many ESP schools had only one or two reading teachers remaining from the ESP. However, these teachers tended to be the most dedicated who had adopted the methodology. Thus, the Reading component findings reflect results achieved under the Spanish Language component and are representative of the project achievements. The Math component was greatly appreciated by the schools, but was not a key focus of the evaluation given that USAID’s education strategy and future programming prioritize reading. Findings, Conclusions and Recommendations 7 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation Question 1: Achievement of Results Q1: DID THE PROJECT MEET ITS EXPECTED RESULTS AND TARGETS AS DESCRIBED IN THE COOPERATIVE AGREEMENT? Findings Expected results (IR1-IR3). Per the cooperative agreement, the program was ultimately expected to train 5,500 teachers to increase effectiveness in the first cycle of public primary education and improve 150,000 students’ performance in Mathematics and Spanish Language in 400 schools. The program was designed to improve management skills of 400 school principals and 150 district technicians as well as educate 98 parent schools and 150 Parent-Teacher Associations (PTAs) on their roles and functions. As illustrated in Table 5, the program exceeded its targets for all beneficiary groups for IR1 New Education Management Structure Strengthened, IR2 Literacy Learning Improved, and IR3 Math Learning Improved. The only exception was School Boards due to changes in the Ministry’s education strategy, which prevented the program from continuing work. Table 5. Five-Year Results for Spanish, Math and Education Management Result Target Actual Math and Spanish Language teachers trained 5,500 10,294 Students’ learning improved in Math and Spanish Language 150,000 284,000 Schools participating in ESP 400 1,837 School principal management skills improved 400 2,806 District technicians trained 150 612 Parents schools trained 98 114 PTAs trained 150 212 School Boards trained 150 124 Source: ESP monitoring data The ESP aimed to improve reading and writing competencies for 89,500 children. According to ESP data, the actual number of children with improved skills was 131,033, thereby exceeding the target. This contribution to students’ learning was measured by the percentage of students with statistically significant improvement in Math and Reading Comprehension in comparison to the performance of the control group (IR2 Result 1: Competencies for critical reading of writing of first to fourth grade students improved). The Monitoring and Evaluation component conducted evaluations of student learning in math and reading comprehension in third and fourth grade each year of the project. Results (Valverde, 2014) showed that on average students with at least one year in the program performed better than their peers in the control group; those with two consecutive years in the program gained even more percentage points than students in non-intervention schools. Independent analysis of the ESP data on third and fourth graders indicates that while fourth graders in the comparison group improved by 6.5 percentage points on average in reading comprehension, the fourth graders with two years in ESP improved by 8.9, 9.6, and 10.8 percentage points in 2011, 2012, and 2014, respectively. The difference between the improvement observed in the comparison group (6.5%) and ESP (10.8%) was 4.4% in 2014 and the improvement observed in ESP students was statistically significant. In terms of qualitative data from the performance evaluation, the majority of teachers and principals interviewed stated that Findings, Conclusions and Recommendations 8 students with continuity in the program improved their literacy skills significantly compared to students from other schools that transferred or who were taught by untrained teachers. The program also measured changes in teachers’ competencies in Math and Spanish Language. During interviews with project coordinators, the evaluators learned that the evaluation methods for Math and Reading teachers differed. For the Math component, the project assessed teachers’ performance using the same first grade Math test given to students. A pre-test and post-test was applied to measure competencies before and after the training intervention. To measure Spanish Language teachers’ competencies, the program developed a criterion-referenced classroom observation form and conducted assessments three times a year. The results were kept in a portfolio and teachers received a final score at the end of the year. The program improved competencies of 950 teachers exceeding the target (900) in the PMP. Dissemination of teacher and student performance assessments. The midterm evaluation (recommendation #6) stressed the need to conduct, analyze and disseminate teacher and student performance assessments. Key informant interviews with MINERD staff, principals and teachers revealed that the dissemination of student and teacher evaluation results varied at each level of the ministry. At the national level, the Vice Minister of Quality Assurance and the Evaluation Director had received the results of the student evaluations. The Basic Education Department for the first cycle of primary education did not received the results of the student or teacher performance evaluations. At the district level, the regional and district directors interviewed in Jarabacoa, Santiago and La Vega attended a presentation of the student achievement results, but were less informed of the teacher assessment results. In the 53 schools visited, 78% of principals and 21% of teachers received the results of teacher and student performance evaluations. The majority of principals (75%) attended a dissemination meeting where results of the Math and Reading assessments were presented by district but not disaggregated by school. Thus, the principals were not informed about their students’ performance. Less than one-third of teachers (27%) received the results of their students’ performance. They asserted that the results arrived after the students had been promoted; thus, it was too late to address any difficulties. Teacher assessments from classroom observations were disseminated to school principals and teachers through a signed copy of the observation feedback and debriefing meetings. Gender violence awareness created (IR5). This result was executed through the Safe Schools intervention. The result indicator was defined in the PMP (2014) as “significantly contribute to violence prevention in schools as well as provide the necessary mechanisms to support and refer cases detected in schools and their environment.” This component was added in 2009 with a small pilot activity in 2010, but was not fully implemented until 2013. The SS component provided gender-based violence awareness training to administrative and technical staff, students, teachers and community leaders. ESP identified 42 cases of school-based violence in 25 schools; most of the cases were physical or psychological abuse. There was insufficient time to develop sustainable formal mechanisms to address all cases and to make a significant impact on preventing school violence in all target schools. However, the project tracked all cases of abuse identified and worked with prosecutors to pursue redress. ESP conducted a pre- and post-intervention survey to measure teachers’ and students’ perceptions of violence and found there was a significant reduction in all indicators of violence in Santiago and Santo Domingo schools. The greatest improvements were in Santiago; over 80% of teachers stated violent incidents had improved before school, during recess, after school, and in the classroom. In Santo Domingo, between 50 to 68% of teachers observed similar improvements. Strengthen technical capacity to educate children with special needs (IR6). This result was implemented through the Inclusive Education component. According to the PMP, the expected result was “to support MINERD to develop the capacity in Dominican Republic public schools to detect, diagnose, treat and support school aged children with physical or psychological disabilities.” The project Findings, Conclusions and Recommendations 9 planned to work in 100 schools originally. The target was reduced to 55 given the short timeframe and limited resources. USAID and PUCMM agreed that this was a more effective approach. The project targeted 500 children for proper diagnosis and adequate treatment and support. Using a pre-populated list of students with special needs and a diagnostic instrument provided by MINERD, the project assessed 553 students in 55 schools using the EVALUA instrument. The results of the diagnostic assessment revealed that 272 had classified disabilities according to Ministry definitions. ESP found that nearly 20% of students in each school had special needs. The majority of children diagnosed with learning disabilities (161 children) were found to have cognitive challenges, which required long-term interventions. Due to the short timeframe of the project and lack of human resources and physical space to attend to severe cases, the project addressed short-term medical needs. The program treated 50 of the 272 cases diagnosed, representing 18% of the total. PUCMM garnered support from medical professionals who volunteered their services and from clinics that provided services at a reduced cost. By the end of the project, students had benefited from 318 medical and lab exams, two surgeries, and 48 donated pairs of eyeglasses. ESP further developed a special needs specialization program. Twenty students were funded through USAID. ESP, with financing from INAFOCAM, provided an additional 300 scholarships to address the critical shortage of special education teachers. (See Questions 4 and 6 for more details.) Conclusions Below is a summary of findings and conclusions for question 1. Findings Conclusions Finding #1: The program exceeded its five-year targets under IR1 New Education Management Structure Strengthened, IR2 Literacy Learning Improved, and IR3 Math Learning Improved. Conclusion #1: The program has achieved its targets for five-year expected results outlined in the cooperative agreement. Finding #2: Fourth graders with two years in ESP improved in reading comprehension by 10.8 percentage points in 2014. The difference between the improvement observed in the comparison group (6.5%) and ESP (10.8%) was 4.4% in 2014 and was statistically significant. Conclusion #2: Student achievement results indicate that ESP has met its objective of improving student performance in Spanish Language. 4 Findings #3: 90 schools participated in the Safe Schools intervention. The SS conducted trainings, identified and tracked 42 cases of violence in 25 schools, and worked with prosecutors to pursue redress. 80% of teachers in Santiago and roughly 60% of teachers in Santo Domingo reported a reduction in violent behavior. Conclusion #3: The SS intervention was initiated as a pilot without results targets; the project achieved an important first step by raising consciousness about GBV and reducing incidences of school-based violence in schools. Finding #4: The project properly diagnosed 227 students in 55 schools, and treated 50 students through provision of two surgeries and 48 pairs of eyeglasses; short-term interventions were implemented due to time and resource constraints. Conclusion #4: The IE component identified a large number of students with learning disabilities that required long-term interventions. ESP was unable to fulfill its targets for treatment and adequate support due to the short timeframe and lack of human and physical resources to support long-term interventions. Finding #5: Results of student evaluations were disseminated to the MINERD Vice Minister of Quality Assurance and Evaluation Director, district and regional Conclusion #5: Student learning assessment results were generally disseminated at all levels of MINERD. The data disseminated to schools reflected district- 4 Please note the evaluation team did not analyze student Math achievement data. Findings, Conclusions and Recommendations 10 directors, and principals; 70% of schools reported not receiving student-level data or school performance data. 27% of teachers received student achievement data, but it arrived too late to address difficulties. Finding #6: Teacher evaluation results were disseminated to school principals and teachers through observation records and debriefing meetings. wide results, and thus were not disaggregated by student to inform teacher instruction or by school to support school management decision-making. Recommendations Recommendation #1: Increase the timeframe for implementing the SS and IE components; set realistic targets. Based on: Conclusion #3: The SS intervention was initiated as a pilot without results targets; the project achieved an important first step by raising consciousness about GBV and reducing incidences of school-based violence in schools. Conclusion #4: The IE component was unable to fulfill its targets for treatment and adequate support due to the short timeframe and lack of human and physical resources to support long-term interventions. Recommendation #2: Incorporate school performance data into leadership trainings and student achievement data into teacher trainings in order to directly improve school performance. Recommendation #3: Disseminate teacher and student results of performance evaluations to the district and national MINERD (e.g., Basic Education department for first cycle of primary education). Based on: Conclusion #5: Student learning assessment results were generally disseminated at all levels of MINERD. The data disseminated to schools reflected district-wide results; and thus were not disaggregated by student to inform teacher instruction or by school to support school management decision-making. Evaluation Question 2: Reading Results Q2: HOW DID THE ESP PERFORM IN TERMS OF IMPROVING READING SKILLS OF PRIMARY SCHOOL BOYS AND GIRLS IN TARGETED COMMUNITIES? WHAT FACTORS ARE CAUSING READING SCORES TO REMAIN LOW OVERALL, EVEN IN CASES WHERE PROGRESS WAS MADE UNDER ESP? Findings Reading performance results. The ESP quantitative data on student reading performance for first and second graders were limited because there is no established methodology for testing reading comprehension for first and second graders in the region, according to a PUCMM statistician. However, the ESP quantitative data on reading performance for third and fourth graders were revealing. ESP administered an end-of-year test to third and fourth graders that included 27 common reading comprehension questions, which made it possible to observe improvement for students progressing from third to fourth grade in ESP, as well as for a nationally representative comparison group of students not in ESP. Independent analysis of these data on third and fourth graders indicates that while fourth graders in the comparison group improved by 6.5 percentage points on average in reading comprehension, the fourth graders with two years in ESP improved by 8.9, 9.6, and 10.8 percentage points in 2011, 2012, and 2014 respectively. On average, ESP students performed 3.3% higher than the comparison group. The percentage increased each year of the program. As the table below indicates, ESP students outperformed the comparison group by 2.5% in 2011, 3.2% in 2012, and 4.4% in 2014. The difference between the improvement observed in the comparison group and the improvement observed in ESP students was statistically significant in 2014. Findings, Conclusions and Recommendations 11 Table 6. Reading Performance of Third and Fourth Grade Students Group N Years in ESP Calendar Years Ave. 3rd Grade Score (%) Ave. 4th Grade Score (%) Ave. % Point Improvement from 3rd to 4th Grade Difference from Comparison Group 1 157 0 2010-2011 39.4 45.8 6.5** (2.7) N/A 2 1,712 2 2010-2011 41.6 50.4 8.9*** (17.9) 2.5 (1.4) 3 2,193 2 2011-2012 41.3 50.8 9.6*** (18.9) 3.2 (1.6) 4 3,837 2 2013-2014 43.9 54.7 10.8*** (29.2) 4.4* (2.3) Note: t statistics in parentheses. * p < 0.05 ** p < 0.01 *** p < 0.001 On a national level, however, third and fourth grade reading scores are still quite low. In 2014, third grade students achieved an average of 58% of the competencies in reading comprehension, with scores ranging from a low of 41% to a high of 77%. In fourth grade, the results are lower, averaging 48% and ranging from 30 to 70%. The Second Regional Comparative and Explanatory Study (SERCE) and the Third Regional Comparative and Explanatory Study (TERCE) are the first two regional assessments of student achievement comparable across time. According to the American Institutes for Research (2015), these assessments show that the Dominican Republic was the lowest-performing country in reading in the region, but the Dominican Republic also demonstrated the most improvement in third grade reading and among the most improvement in sixth grade reading in the region between 2006 and 2013. Teachers interviewed perceive an average increase of 65% in the proportion of students able to read fluently with comprehension in grades 1-4 since the beginning of the project. According to reading teachers interviewed, the greatest gains in reading achievement based on formative assessments were observed in third grade, where there is now less repetition and nearly 90% can read and write fluently. This finding is consistent with ESP test results. However, classroom observations found that student performance decreases in fourth grade for three key reasons: influx of transfer students, lack of materials aligned with the curriculum, and insufficient mentoring. Teachers reported that diagnostic tests showed that ESP students outperformed newly transferred students. Therefore, transfer students who have not been taught with the ESP methodology tend to affect the overall performance of the cohort. During classroom observations, evaluators observed that fourth grade students appeared less enthusiastic towards reading compared to students in grades 1-3. Teachers attributed the lackluster motivation to lack of textbooks or workbooks to engage students in reading activities. Teachers stated that unlike for grades 1-3, they do not have sufficient materials (e.g., workbooks, teacher guides, grade￾appropriate library books, diagnostic/formative assessments) specifically designed for fourth grade. Due to lack of textbooks, teachers write stories on large flipchart paper, which is not visible for all students, especially in overcrowded classes. As a result, only a few students are engaged in reading the text. Moreover, students who have already been exposed to the library reading books in the third grade express less interest in reading compared to newly transferred students who are excited by the reading corners. Hence, teachers recommended continuity of the program and materials in fourth grade and beyond. Further affecting student achievement is the finding that fourth grade teachers received less frequent coaching. According to teachers interviewed, reading coaches tended to concentrate coaching in the first three grades. In one particular case, a second grade teacher replaced a fourth grade teacher randomly selected for the observation. The principal explained that the fourth grade teacher was Findings, Conclusions and Recommendations 12 “trained,” but the second grade teacher was a “PEF teacher.” His definition of a PEF teacher is a teacher who received intensive coaching and who had adopted the reading methodology. Factors associated with high reading achievement. Ranked in order of most frequent response by MINERD staff, teachers and principals, the factors most associated with high reading achievement are: (1) use of diverse teaching and learning materials aligned with the curriculum, (2) well￾trained teachers, (3) application of an effective reading methodology, and (4) parental involvement in reinforcing reading. The training and intensive mentoring built teachers’ competencies and confidence to deliver high quality instruction using diverse materials and activities to create dynamic classrooms. Teachers use textbooks, storybooks, pictures, newspapers, fables, cartoons and real-life examples to teach reading comprehension. Teachers and principals stated that the innovative and effective methodology leads to high reading achievement. They claim the effective ESP methodology highly motivates teachers and students. It relates to daily life, builds off of prior knowledge and produces visible results. The reading corners, library books and displays (print-rich environment) were also found to significantly contribute to high reading achievement. In model classrooms where teachers successfully implemented all strategies and children’s storybooks and library books, the majority of students could read aloud and respond to comprehension questions, as early as first grade. Parental involvement – in the reading fairs, volunteering to read in the classroom or with their child at home – was found to significantly contribute to reading achievement. Peer learning (e.g., grouping by ability) and sustained reading time in the library or classroom were also considered important. Finally, continuity in the program was deemed critical for building on students’ prior knowledge and ensuring they advance in their reading, writing and critical thinking skills. Factors associated with low reading achievement. According to teachers, principals and MINERD respondents, the key factors that contribute to low reading achievement in order of priority are: (1) lack of parental reinforcement, (2) family problems, (3) learning difficulties/disabilities, (4) discontinuity of methodology and (5) teachers' insufficient competencies. Nearly all respondents stated that lack of parental involvement in reinforcing the reading methodology is the number one factor affecting reading achievement. According to principals and teachers, parents do not understand the ESP methodology; they are “stuck in the traditional phonics approach” (e.g., using the Nacho book) and believe the assigned homework is too difficult. Parents who attended the reading fair and witness first graders reading stories fluently have changed their expectations. However, parents from low￾socioeconomic backgrounds with limited education are more difficult to reach. Their involvement in school meetings and events is limited due to transportation barriers or work conflicts. The second greatest factor contributing to low achievement was family-related problems (e.g., drugs, violence, abuse, poverty, neglect). Many low-performing children live with their grandparents or other relatives who do not take responsibility or genuine interest in their child’s learning. This diminishes students’ interest and motivation to read. Additionally, students with learning difficulties or disabilities that have not been addressed by a psychologist or counselor lag behind. Finally, discontinuity of the methodology due to teacher transfers or promotion to fifth grade also leads to low reading achievement. ESP reading methodology. Aligned with the MINERD philosophy to literacy instruction, the ESP adopted a communicative and functional approach to implement the literacy curriculum. It is a blended approach that emphasizes de-codification of syllables, sounds and meaning within authentic or student￾generated text. It relies on materials and familiar concepts commonly found in the environment to build on students’ prior knowledge. For instance, the literacy curriculum begins with teaching the name of the child (which helps build self-esteem). The teacher shows the child cards with his or her name and asks the child to write what he or she sees. This is the first step in the four stages of language acquisition. In this pre-syllabic stage, students begin to associate spoken words with characters (phonemic awareness). In the second stage (syllable phase), they can read and write syllables, but not yet identify letters. Unlike English, in Spanish many sounds represent a syllable; thus, the syllable phase precedes letter Findings, Conclusions and Recommendations 13 identification. In the third stage – syllable-alphabetic stage – students begin to recognize letters. They can write one- and two-letter syllables, but not fluently. In the fourth alphabetic phase, students can spell words correctly and differentiate among words, syllables and letters. ESP also teaches three levels of reading comprehension – literal, inferential and critical. The literal stage consists of factual comprehension questions in which the answers are found within text. In the inferential stage, students are asked to interpret the text and infer the response. In the third phase, critical comprehension, students adopt a position (e.g., what did you like, do you agree/disagree) or provide a critique. Children learn all three phases of reading comprehension from first grade and are expected to read with comprehension by the end of first grade. Exposing children to whole words and stories early has produced dramatic results. In Puerto Plata, an exemplary first grade teacher who uses a variety of children books, interactive games and classroom management strategies to fully engage children reported that 33 of her 35 students could read fluently with comprehension. Classroom observations confirmed that her students in first grade could read short stories, respond to comprehension questions, and write a critical reflection. In Jarabacoa, a teacher who formally taught only phonics has created a remedial class for overage students and groups them by ability. Knowing the stages and using the differentiated learning kits from PUCMM has allowed her to advance her students. In her regular third grade class, the teacher indicated, “Over 90 percent can read thanks to the ESP methodology.” Many teachers stated that knowing the stages of reading and writing acquisition and having the tools and resources to track students’ progress and intervene when necessary has significantly improved results. Eighty percent of 49 teachers interviewed have students with special needs in their classroom, 70% provide individualized instruction and nearly 30% prepare differentiated tasks sometimes with support from the special needs coordinator, the principal or a school psychologist. Teachers and principals are convinced that the ESP reading methodology produces results. The majority of respondents stressed that students in the lower grades are better able to express themselves, are more creative, and have higher critical thinking skills when compared to students in higher grades or non-ESP schools. Challenges to sustainability. A key threat to sustainability of the reading progress achieved under ESP is reverting to the old methodology when trained teachers are transferred and replaced by an untrained teacher or when students advance to fifth grade. Having witnessed the loss of momentum in fifth grade, several schools in Santo Domingo have shifted ESP-trained teachers from fourth to fifth grade. While this appears like a viable solution for replicating the program with existing resources at the school level, PUCMM experts do not recommend this approach since these teachers are not trained to teach the second cycle curriculum. Furthermore, it is a loss of USAID’s investment if they are not retained in the lower grades. Some schools with high teacher attrition have requested PUCMM coaches to provide an orientation to new teachers, while existing ESP teachers or coordinators provide support with using the materials. PUCMM in collaboration with MINERD has created Teacher Coordinators in each school trained to mentor new teachers in implementing the methodology. Though the evaluation team only found a few instances where the mentors exist, it is a promising option for sustaining the program. Conclusions Conclusions for Question 2 and their associated findings are shown below. Findings, Conclusions and Recommendations 14 Findings Conclusions Finding #1: Fourth graders in the comparison group improved by 6.5 percentage points on average in reading comprehension. Fourth graders with two years in ESP improved by 8.9, 9.6, and 10.8 percentage points in 2011, 2012, and 2014 respectively. Overall, ESP performed 3.3% higher than the comparison group. The percentage increased each year of the program from 2.5% in 2011 to 3.2% in 2012, and 4.4% in 2014. The difference between the improvement observed in the comparison group and the improvement observed in ESP students in 2014 was statistically significant. Conclusion #1: The ESP has made significant strides in improving students' reading skills in the early grades through provision of diverse teaching and learning materials, well-trained teachers, and application of an effective reading methodology. Results are evident in classroom observations and test results. Finding #2: Teachers report that the greatest reading gains, based on formative assessments, have been observed in third grade while performance decreases in fourth grade. Finding #3: Classroom observations and interviews indicated fourth grade teachers received inadequate materials and coaching. Conclusion #2: Additional materials (workbook, teacher-guides and grade-appropriate library books) are needed to sustain improvements in fourth grade and beyond. Finding #4: Use of diverse teaching and learning materials aligned with the curriculum, well-trained teachers, application of an effective reading methodology and parental involvement in reinforcing reading are factors associated with high reading achievement. See Conclusion #1 Finding #5: According to teachers, principals and MINERD staff interviewed, factors that contribute to low reading achievement include lack of parental reinforcement, family problems, learning difficulties, discontinuity of methodology and teachers' competencies. Conclusion #3: The most significant challenge contributing to low reading achievement is that parents do not reinforce reading at home and they lack understanding of the new methodology. Finding #6: Communicative and functional literacy methodology is a blended approach that emphasizes phonemic awareness, phonics and comprehension (de￾codification of syllables, sounds and meaning within authentic text). See Conclusion #1 Finding #7: 70% of teachers provide individualized attention and 30% differentiate tasks for special needs students. Conclusion #4: The majority (70%) of teachers require additional support to develop differentiated tasks for special needs students. Finding #8: Teacher transfers and promotion to 5th grade threaten the continuity of student achievement. Conclusion #5: Teacher transfers and discontinuity in the second cycle hamper the sustainability of the program. Recommendations Recommendation #1: Continue providing support in the ESP schools through building the capacity of Teacher Coordinators to mentor new and existing teachers. Based on: Conclusion #1: The ESP has made significant strides in improving students' reading skills in the early grades through provision of diverse teaching and learning materials, well-trained teachers, and application of an effective reading methodology. Results are evident in classroom observations and test results. Recommendation #2: Provide Teacher Guides, textbooks, grade-appropriate library books, and diagnostic/formative assessment booklets for fourth grade aligned with the curriculum. Based on: Conclusion #2: Additional materials (workbook, teacher-guides and grade-appropriate library books) Findings, Conclusions and Recommendations 15 are needed to sustain improvements in fourth grade and beyond. Recommendation #3: Train parents in the reading methodology and in how to support their children's reading at home and in school; engage more parents in reading fairs and school-wide events through offering incentives and convenient schedules. Based on: Conclusion #3: The most significant challenge contributing to low achievement is parents do not reinforce reading at home and they lack understanding of the new methodology. Recommendation #4: Provide more tools, games and manipulatives to attend to special needs students (e.g., phonics exercises, magnetic letters, big books, CDs, videos, workbooks). Based on: Conclusion #4: The majority (70%) of teachers require additional support to develop differentiated tasks for special needs students. Recommendation #5: Extend the program to the second cycle (to at least sixth grade) and sustain innovation circles to promote transfer of knowledge among new and well-trained teachers; provide demonstration classes with live students and model teachers or videos of exemplary classes to illustrate to new teachers the power of the methodology.5 Recommendation #6: Provide compensation or special recognition to model teachers who replicate the program in their school or zone. Based on: Conclusion #5: Teacher transfers and discontinuity in the second cycle hamper the sustainability of the program. Evaluation Question 3: Safe Schools Results Q3: HOW DID THE SAFE SCHOOLS ACTIVITY PERFORM FOR BOYS AND GIRLS IN TERMS OF IMPROVING SAFETY AND CREATING AN ENVIRONMENT OF TOLERANCE IN TARGETED SCHOOLS? Findings In the early phases of the program, ESP detected an alarmingly high incidence of violence in primary schools. USAID subsequently introduced the Safe Schools component in 2013. Through an exploratory study led by PUCMM on teachers’ and students’ perceptions of violence, ESP systematically identified that violence occurred most frequently during walks to and from school, in classrooms, and during recess. School principals, teachers, parents and, to a lesser extent, community leaders discussed and developed strategies to resolve the issues. In addition, ESP conducted six workshops to specifically train teachers on how to monitor recess and to provide students with structured activities. As a result of ESP interventions, respondents highlighted several improvements in the school-learning environment: (1) reduced bullying and fighting, (2) reduced use of abusive language, (3) development of codes of conduct in schools, and (4) improved classroom management. These improvements are encouraging. However, evaluation team members witnessed eight incidents of school violence largely among younger students in grades 2-4. Additional violent and disruptive behavior observed included schoolyard bullying, disrespectful student behavior towards principals, and unruly classrooms and hallways. The improved behavior and remaining concerns are highlighted below. 5 Model teachers were observed in Puerto Plata (Albuerge de Martina, first grade), Jarabacoa (Dulce Maria Tiburcio Rincon, third grade), and Santo Domingo (Hermanas Mirabal, second grade). Findings, Conclusions and Recommendations 16 Improved student behavior. The most significant changes observed by 60% of teachers and principals interviewed were better-disciplined students, which led to a reduction in fighting. According to teachers and principals interviewed, youth who participated in the SS activities had increased awareness of violence and bullying as inappropriate, intolerable behavior that negatively impacts the teaching and learning environment. These SS-trained students tended to be more respectful and were more engaged in classroom activities. Focus group discussions with youth indicated that students involved in the SS activity were better able to identify violent actions and develop conflict resolution strategies for self-protection compared with those who had not participated. As a result of increased awareness of violent behavior and improved self-control, there have been fewer occurrences of fighting. School principals reported that in the past fighting occurred between two to three times a week. Though it still occurs, it is less frequent. Factors contributing to a reduction of fighting included increased teacher supervision during recess and less tolerance of fighting. The Education Management and Spanish Language components also contributed to strengthened school leadership and improved classroom management. Reduced bullying. Forty percent of SS respondents reported a reduction in bullying. This outcome was attributed to improved supervision during recess, increased awareness among teachers of unsafe areas due to school mapping exercises, and students’ improved self-esteem. While school principals and teachers reported a reduction in bullying, focus groups with 179 youth indicated a need for continued vigilance. About half of students interviewed reported feeling threatened by older or physically larger students and being bullied frequently. For instance, older boys and girls steal school supplies, snacks or lunch food as leverage to demand favors. Some bullies have charged students a fee to enjoy the playground. Reduced violence and remaining challenges. Eighty percent of respondents involved in SS activities reported a reduction in violent behavior. However, observations indicate that violence is still prevalent in schools in low socioeconomic, crime-ridden, and drug-infested communities. The evaluation team observed eight violent fights in several schools in Santiago. The fights were between students of both sexes with older boys typically being the perpetrator. All instances of physical violence occurred within the presence of evaluation team members and primarily during recess on both supervised and unsupervised playgrounds. Five cases required medical attention. For instance, one rock-throwing incident resulted in a serious eye injury and rush to an emergency facility. In these schools, fighting on school grounds, disruptive behavior in the classroom, and a constant drone of noise were the norm. Tolerance of harmful, disruptive and unruly behavior poses serious threats to students learning and safety. Due to fear of their child’s safety, 25% of parents interviewed reported that they visit the school at least three times per week. Many youth have witnessed forms of violence in the school and community. Nearly half of students interviewed (49%) had observed aggressive and violent behavior (e.g., hitting, spitting, slapping, kicking, scratching with fingernails or safety pins) on school grounds. Drug dealing and murders have occurred within the proximity of some schools or within students’ communities. Two schools in Santo Domingo and Santiago claim that the violence situation is getting worse. Use of abusive language. While 73% of respondents claim that use of abusive language has been reduced, 82% of students reported that name-calling and offensive language persists. Students reported that the use of unfavorable nicknames is common among students. Such behaviors were shown to have serious consequences for students’ safety, health, and emotional wellbeing. A young student was observed experiencing an anxiety attack after having been “called a name,” and required emergency medical attention. According to the SS Coordinator interviewed, this was because the student was still in shock from a recent drive-by shooting of two former students that had occurred across the street from the school. During other school visits, teachers reported that girls who had been raped or abused Findings, Conclusions and Recommendations 17 were also at risk of emotional abuse. One school that had received the SS training proactively prepared students for how to deal with a formally abused student returning to school. While bullying and abusive language have improved in schools where students had learned that “words can hurt,” in the majority of schools, principals reported that it was difficult to change the language and behavior due to external influences, such as the “machismo culture,” physical violence in the home, and social media. Conditions contributing to improved safety and behavior. Despite the challenges observed in many schools visited, there were numerous instances of exemplary schools with peaceful and respectful learning environments. Actions taken to create a positive learning environment included creating policies that promoted a culture of peace, values and respect; peace agreements signed by students; and codes of conduct in every classroom. School- or district-wide interventions, involving all actors (e.g., students, teachers, parents) had the most impact on sustaining the new culture. In one district in Jarabacoa in which all schools participated in the SS intervention, every school established codes of conduct with student participation. The district-wide intervention supported by a critical mass of principals led to significant improvements in student conduct. A school-wide policy that had a significant impact on reducing bullying and fighting was developing activities during recess. During the Math trainings, teachers informed PUCMM that school-based violence during recess was affecting students’ learning in the classroom. Teachers requested ESP to provide training on activities to conduct during recess. In several active and responsive schools that have implemented the suggested strategies, violence has been reduced. A school in Puerto Plata separated recess periods for lower grades (1-5) and upper grades (6-8), to prevent older students from bullying younger students. The school also engaged youth in designing games and activities to keep students entertained during recess. This strategy has eradicated bullying in the school. Schools with sustained participation of all stakeholders in the program exhibited the greatest results. For example, schools with principals enrolled in leadership certificate training, parents engaged in parent schools, and teachers participating in the literacy training program had a peaceful learning environment even in the most difficult areas. A school in Santo Domingo, for example, that located in a high-risk area demonstrated strong classroom management, leadership and parental involvement. In this particular case, both the teacher and principal had participated in the certificate-training program and parents had participated in a seven-month long program about how to support their children at home. Parental involvement is another very important factor. Among 165 parents interviewed, there were 15 different groups involved in school affairs. Such organizations included: School Boards, PTAs, Class Committee, Mothers’ Club, Neighborhood Committee, Society/Association of Parents and Friends, Mothers and Fathers in Action, Parents’ School Group, Church Committee, Civil Society Committee, Neighborhood Cultural Group, Prayer Group, and Homemakers of Merced. Parents volunteer time, talent, and skills by organizing and leading aerobics classes, painting classes, and woodworking classes for new, extended day schools so that students “will not be bored with the extra hours at school.” In a model SS, República Venezuela, in the Santiago region, parents were so encouraged by the school principal’s decision to establish a “Culture of Peace” that they organized the donation of comfortable furniture to “make the school like a home.” Parents also informed of their concerns regarding GBV and sexual awareness. Eighty percent of the 165 parents interviewed recommended that age-appropriate sex education be taught in the schools as a means to secure the safety of boys and girls and to “lower fear” in and out of school. Only one parent explicitly stated, “I will be the one to teach my daughter about sex.” When asked at what grade to introduce the topic of sex education, a group of parents agreed that “starting with the first grade using material and language appropriate for small children,” was not too soon to begin. Findings, Conclusions and Recommendations 18 Improvements in academic performance. In SS schools visited, 44% of respondents interviewed reported improvement in academic achievement as a result of ESP, with 31% noting an improvement in reading specifically. Only a few teachers and principals noted that academic achievement had improved as a direct result of safe schools activities. For instance, one principal asserted, “When the teacher has to spend less time on classroom management, there is more time for children to learn.” In cases where academic achievement increased due to an improved learning environment, it improved in all subjects, not only reading. Increased reading achievement was largely attributed to the effective reading methodology and classroom libraries. Yet, the SS component was credited with contributing to positive attitudes, improved concentration and behavior in the classroom. Conclusions A summary of findings and corresponding conclusions are presented in the table below. Findings Conclusions Finding #1: The most significant changes observed by 60% of teachers and principals interviewed were better-disciplined students and a reduction in fighting; 40% also reported a reduction in bullying. Finding #2: 80% of respondents reported a reduction in violent behavior; however, observations indicate that violence is still prevalent in low￾income/high crime areas. Finding #3: 73% of respondents claim that there is less abusive language, though 82% of students report it is still a problem. Conclusion #1: There is evidence of improved sensitization of gender-based violence and less bullying, yet violence and bullying persist. Conclusion #2: School environments and cultures that tolerate harmful, disruptive and unruly behavior pose serious threats to students’ learning and safety. Schools with traumatic forms of violence can cause fear, anxiety, and more physical aggression between students. Conclusion #3: The broader 'machismo' culture and abuse in home and community influence student behavior and language. Finding #4: Establishing codes of conduct and promoting a culture of peace, values and respect had a noticeable impact on the learning environment. Finding #5: Teachers monitoring recess, separating lower and upper primary students, and involving youth and teachers in leading productive activities during recess had a positive impact on reducing bullying and fighting. Finding #6: Schools with sustained participation of all stakeholders in the training program exhibited the greatest results. Conclusion #4: In order to create a safe school, there needs to be transformational change in the school. School-wide norms and regulations involving all actors (e.g., principals, parents, teachers, youth) were found to have a noticeable impact on creating a safe and positive learning environment. Conclusion #5: Improving supervision during recess and outside of the classroom can have a positive effect on classroom behavior. Conclusion #6: Long-term training in leadership, parenting, and classroom management are essential for creating a school-wide culture that promotes peace and respect. Finding #7: 44% of respondents reported improvement in academic achievement, as a result of ESP, with 31% noting an improvement in reading specifically. However, the improvement was not solely attributed to SS interventions. Conclusion #7: While the SS component cannot be directly correlated with reading achievement, it did contribute to improved classroom conduct. Recommendations Recommendation #1: Provide SS training for all primary grade teachers, principals and parents. Recommendation #2: Address violence and bullying on a national scale (e.g., student code of conduct handbooks, enforcements, continuous training for all teachers, school principals and parents on roles and responsibilities in creating a culture of peace). Based on: Conclusion #1: There is evidence of improved sensitization of gender-based violence and less bullying, yet violence and bullying persist. Findings, Conclusions and Recommendations 19 Recommendation #3: Collaborate with MINERD and other relevant ministries (e.g., Ministry of Women) to define SS interventions that address GBV, sexual abuse, and other forms of violence based on the socio-cultural context of the DR. Based on: Conclusion #2: School climate and culture that tolerate harmful, disruptive and unruly behavior poses serious threats to students learning and safety. Conclusion #3: The broader 'machismo' culture and abuse in home and community influences student behavior and language. Recommendation #4: Provide training for school personnel to understand the connection between violence and abuse, its impact on student achievement and how it is evidenced in undisciplined behavior. Recommendation #5: Share best practices from model ESP schools that successfully changed school culture and climate to one that promotes peace, responsibility, leadership, and gender equity in order to inform the strategy and training. Based on: Conclusion #1: There is evidence of improved sensitization of gender-based violence and less bullying, yet violence and bullying persist. Conclusion #4: School-wide norms and regulations involving all actors (e.g., principals, parents, teachers, youth) have a sustainable impact on creating a safe and positive learning environment. Conclusion #5: Improving supervision during recess and outside of the classroom can have a positive effect on behavior in the classroom. Recommendation #6: Provide advanced training options for school principals and teachers; implement parent education programs in schools with low reading achievement and/or high rates of violence. Recommendation #7: Tap the rich resources of schools around the globe that have successfully addressed school bullying; provide models and book collections that depict “safe school behavior, attitudes, and skills” as well as strategies to cope, manage, and avert such behavior. Based on: Conclusion #6: Long-term training in leadership, parenting, and classroom management are essential for creating a school-wide culture that promotes peace and respect. Conclusion #7: While the SS component cannot be directly correlated with reading achievement, it did contribute to improved classroom conduct. Evaluation Question 4: Special Needs Results Q4: HOW DID THE ESP PERFORM IN TERMS OF MEETING THE REQUIREMENTS OF TARGETED SPECIAL NEEDS BOYS AND GIRLS? Findings Key activities: There were four key activities that took place under the IE component: diagnostic assessment, treatment of cases diagnosed, training and sensitization workshops, and the creation of IE specialized degree programs. The results of these activities and the challenges are described in this section. Diagnostic assessment and treatment. During the initial assessment phase, the ESP worked with the Ministry to validate the list of students with disabilities and assisted the Ministry with refining its existing diagnostic assessment instrument to adapt it to the local context. ESP conducted diagnostic evaluations of 553 children in 55 schools. Of those assessed, 272 students were classified as having disabilities as defined by the MOE. The majority of the cases diagnosed (68%) were intellectual learning disabilities, which required external support and long-term interventions. Given the short duration of the IE component and limited resources in the communities (see key challenges below), ESP treated 50 cases, representing 18% of the total cases identified. The program documents report that ESP conducted two surgeries and provided 48 pairs of glasses with contributions from the medical community (see Question 1 for more details). Findings, Conclusions and Recommendations 20 The evaluation team interviewed school staff that had been involved in the IE component to confirm the diagnostic assessments and treatment provided by ESP. Respondents consisted of teachers, coordinators, counselors, psychologists, principals and deputy principals. School staff interviewed in the 24 schools informed that 115 students with disabilities were diagnosed. Schools reported that 38% of disabilities identified were intellectual, 16% were visual, 30% were learning problems, and 10% were physical disabilities. Of the total cases diagnosed, respondents reported that 43% of the cases were treated. The majority of the cases were visual or learning problems. The types of support ESP provided included: referral to a specialist, provision of IE materials, and tutoring. The respondents confirmed that intellectual or physical disabilities were not adequately addressed. Awareness raising. In addition to diagnosing children with disabilities, ESP provided three independent sensitization trainings for over 90 administrators, teachers, students, and community leaders. The first workshop was an awareness-raising workshop for principals and school psychologists or counselors. The second was a special needs curriculum adaptation workshop for school coordinators. The objective of the third workshop was to train teachers who would be appointed to the special needs classroom (Aula de Apoyo). According to evaluation data, one-third of IE respondents highlighted the sensitization training as a key intervention. Stakeholders noted there is increased awareness and motivation to address learning difficulties and disabilities. As a result of the training, teachers now understand that children with disabilities have rights to education and must be supported to learn. Teachers have educated parents on how to support their child’s learning at home. The change in teachers’ and parents’ mindset was described as, “In the past, parents and teachers thought disabilities were an illness that could not be treated; now, they believe that children with disabilities can learn.” IE specialization program. Though not originally part of the ESP cooperative agreement, the program identified the need for increased capacities in inclusive education to address the critical shortage of IE teachers and counselors. Working with the Instituto Superior de Formación Docente Salome Urena (ISFODOSU), ESP established an IE degree program. USAID funded 20 students and INAFOCAM has supported an additional 300 to date to continue the momentum of the program. Key challenges: Over 90% of IE respondents (96%) stated that the program’s activities and duration were insufficient to address all needs identified. There were numerous challenges that impacted this component’s effectiveness. The most significant findings were limited follow-up and time for implementation, insufficient resources, the delayed appointment of a dedicated IE teacher, and a lack of materials, knowledge and information to treat cases. Timeframe for implementation and assessment. As the IE component began in 2013 and culminated at the end of 2014, there was only one year of effective implementation. The majority of time was spent on the assessment phase. ESP conducted laboratory tests and psychological and education assessments that took 4-5 hours to complete per child. Assessors also met with teachers and parents to confirm diagnosis. It took six months to complete the assessment in all schools (January to June 2014). This left only six months to provide training and follow-up assistance. Due to limited timeframe, severe cognitive or physical disabilities could not be fully addressed. Based on interviews with teachers, principals and program staff, ESP focused on short-term interventions providing vision or medical care that did not require long-term follow-up assistance. The program referred schools to specialists within their communities for follow-up. Where there were no existing specialists, schools could not address the critical needs independently and were left feeling unsupported. Findings, Conclusions and Recommendations 21 Limited human and physical resources. A major challenge to sustainability of the IE component is that schools are not equipped with the human or physical resources to attend to students with special needs. Only four of 27 schools visited had a special needs classroom (Aula de Apoyo or Aula de Recursos). In general, schools do not have functioning spaces/classrooms or sufficient resources to support children with special needs. The majority of schools lack a dedicated IE Coordinator to work with students with special needs outside of the classroom or with teachers to develop individualized lesson plans. The Ministry announced that it would appoint a Special Needs Coordinator for the Aula de Apoyo in each ESP school. As the law requires a competitive selection process for recruitment of teachers or staff and there is not an existing pool of candidates, it has delayed the appointment. One school attempted to propose an ESP-trained teacher, but was rejected after going through the selection process and no IE teacher has been appointed there to date. To circumvent the legal issues and bureaucracy halting the appointment, an IE-trained teacher in Santiago who met the profile began intermittently serving as the IE Coordinator. Once the Aula de Apoyo was functioning, the Ministry appointed a new teacher to replace him in the classroom. Lack of materials. Additionally, lack of materials to provide individualized instruction to students is a barrier to implementation. One of the ESP requirements for distributing the IE materials is that there is a Coordinator appointed to the special needs classroom (Aula de Apoyo). As many schools are awaiting the appointment of an IE Coordinator, schools have not received the materials. ESP is housing the materials at CAD (Centro de Recursos para la Atención a la Diversidad, or Center for Attention to Diversity) in the interim. Insufficient information and knowledge to treat cases. The most frequent support requested by teachers was more training on how to detect and respond to learning disabilities. In fact, 70% of teachers claimed the training was too short and general to support them in effectively exercising their IE role. One psychologist noted that the ESP recommendations following diagnosis did not provide specific guidance on follow-up actions. Impact on enrollment and academic performance. Nearly one-quarter of schools reported an increase in enrollment of special needs children as a result of ESP activities. This was largely due to rumors that the school had become more inclusive due to IE interventions. Another explanation is that well-trained teachers encouraged parents of children with disabilities to enroll their child. However, 23% of IE respondents reported that parents do not believe in the schools’ capability to provide for their children’s special needs. For instance, they do not believe that the school can provide assurances that their child will not be bullied or mistreated. With regards to academic performance, 71% of schools reported improvement in general academic achievement of students with special needs; 40% reported improvement in reading achievement specifically. Improved student performance may be owed to teachers giving students more individualized attention, tailored assignments, and having improved expectations. Medical treatment (e.g., provision of glasses, surgery) led to drastic improvements in behavior and performance. However, with regards to reading specifically, classroom observations and interviews indicated that teachers were using the strategies that they had learned during previous reading trainings to identify and support students with special needs. For instance, CETT/ESP teachers received intensive training and mentoring on differentiated instruction, diversity kits for helping struggling readers, and diagnostic assessments for tracking reading performance. The few schools with an IE Coordinator, a classroom dedicated to students with special needs, and MINERD IE materials were beginning to address learning disabilities, but there was little progress with learning and reading achievement. Reading teachers and IE Coordinators informed that the greatest challenge affecting student achievement was lack of follow-up from parents. They stressed the need for greater parental Findings, Conclusions and Recommendations 22 involvement as well as the need for additional training in the IE specialization degree, appointment of an IE Coordinator, and more follow-up assistance from MINERD in order to improve learning achievement of all special needs students. Conclusions Conclusions and supporting findings for Question 4 are presented in the table below. Findings Conclusions Finding #1: ESP diagnosed 272 cases and treated 18% of cases in 55 schools. In the sampled schools visited, ESP diagnosed 117 cases and treated 43% of cases, through short-term medical interventions; intellectual, physical and learning disabilities were not sufficiently addressed. Finding #2: 96% of IE respondents believed the program’s activities were insufficient to address all of the needs identified for students with disabilities. Conclusion #1: There was insufficient time and resources to provide treatment for all children diagnosed with learning or physical disabilities. Conclusion #2: There needs to be a systematic approach for the diagnosis of learning and physical disabilities early in the school year to allow time for detection, intervention, and follow-up assessments. Conclusion #3: Schools need stronger links to clinics and medical professionals for continual diagnosis, referral and treatment. Finding #3: In general, schools do not have functioning IE classrooms, designated IE Coordinators, or sufficient knowledge and resources to detect and support children with special needs; even those with an IE Coordinator and classroom requested additional follow-up support from parents, ESP, and the Ministry to diagnose and treat cases. Conclusion #4: The lack of a dedicated IE Coordinator is severely hindering the schools’ ability to utilize all resources available and address the special needs of all children. Conclusion #5: Schools require additional training, follow-up assistance, and parental support in order to improve learning achievement of children with special needs. Finding #4: 70% of teachers claimed the training was too short and general to support them in effectively exercising their IE role and requested more training to learn how to detect and respond to learning disabilities. Conclusion #6: Training was insufficient to equip school staff and parents with the skills needed to detect, diagnose and treat children with disabilities or learning difficulties. Finding #5: About one-quarter of schools had increased enrollment of children with special needs as a result of learning about the program or outreach from motivated teachers; however, 23% of IE respondents reported that parents do not believe in the schools’ capability to provide for their children’s special needs. Conclusion #7: The majority of schools are not well equipped to attract out-of-school children with disabilities. Finding #6: 71% of schools reported improvement in general academic achievement of students with special needs, with 40% reporting improvement in reading achievement specifically. Conclusion #8: Improved learning in reading is attributed to CETT/ESP reading activities and not IE activities specifically. Recommendations The following are recommendations for future USAID/DR programming to build the capacity of public schools to identify and include children with disabilities. Recommendation #1: Conduct annual diagnostic assessments and develop individualized treatment plans in collaboration with medical professionals in the community. Based on: Conclusion #1: There was insufficient time and resources to provide treatment for all children diagnosed with learning or physical disabilities. Conclusion #2: There needs to be a systematic approach for diagnosis of learning and physical disabilities early in the school year to allow time for detection, interventions and follow-up assessments. Conclusion #3: Schools need stronger links to clinics and medical professionals for continual diagnosis, Findings, Conclusions and Recommendations 23 referral and treatment. Recommendation #2: Rather than aiming to make all schools inclusive, develop model IE schools within zones equipped with all necessary resources. 6 Based on: Conclusion #4: The lack of a dedicated IE Coordinator is severely inhibiting the schools’ ability to utilize all resources available and address the special needs of all children. Conclusion #7: The majority of schools are not well equipped to attract out-of-school children with disabilities. Recommendation #3: Provide IE materials and follow-up support for treatment of cases (e.g., referral lists); continue building the capacity of teachers to engage parents, school staff, and community members in developing and implementing treatment plans. Based on: Conclusion #5: Schools require additional training, follow-up assistance, and parental support in order to improve learning achievement. Recommendation #4: Offer master degrees and specialized certificates in response to demand for qualified IE Coordinators/teachers; concentrate on developing the capacities of teachers who have the disposition and motivation to work in IE classrooms/schools. Recommendation #5: Create support groups for teachers and parents to exchange experiences and successful strategies. Based on: Conclusion #6: Training was insufficient to equip parents and school staff with the skills needed to detect, diagnose and treat children with disabilities or learning difficulties. Recommendation #6: Replicate the CETT/ESP differentiated instruction model with other teachers and schools. Provide intensive training and mentoring on how to develop tailored lessons, conduct diagnostic and formative assessments, and use diversity kits to address reading deficiencies. Based on: Conclusion #8: Improved learning in reading is attributed to CETT/ESP training of reading teachers. Evaluation Question 5: Effectiveness Q5: HOW WELL WAS THE PROJECT IMPLEMENTED, COORDINATED AND MANAGED? WHAT WERE THE STRENGTHS, CHALLENGES AND LESSONS LEARNED? Findings The key strengths, challenges and lessons learned from ESP are highlighted in this section. Key strengths. Fifty percent of key informants lauded PUCMM for its excellent management, training, efficiency and sustained support. School principals were impressed that activities always occurred on schedule per the shared chronogram. The majority of beneficiaries (80%) rated the training as very good or excellent, especially for the Reading and Management components. Nearly 80% found the training 6 This recommendation was proposed by Ecuadorian CONADIS Director, Xavier Torres. The strategies were informed by stakeholders at all levels. Additional suggested strategies included: select schools that are pre-disposed to working with students with special needs; provide training, proper equipment, infrastructure, transportation, and a well-trained psychologist. The schools could serve as a resource center for neighboring schools and a training site. Findings, Conclusions and Recommendations 24 relevant to their work and were able to apply the skills and concepts learned. Teachers and principals were also deeply impressed with the intensity and quality of the coaching. For instance, teachers rated the coaching as very useful, the highest rating on a three-point scale. They consistently noted that coaches visited schools regularly, addressed any issue teachers’ faced through demonstration classes, and provided continual follow-up support. According to USAID and MINERD staff interviewed, the team is highly qualified (e.g., top notch) and has extensive experience working with MINERD, USAID and their respective components. PUCMM has 30 years of experience working with MINERD and 15 with USAID. It was evident in schools visited that past experience on the CETT program, long-standing relationships with teachers, and the utilization of existing CETT models accelerated the program results. For instance, teachers who had been supported by PUCMM since CETT were model teachers; while those with only two years of coaching requested additional support. Due to PUCMM’s fluid communication with USAID and responsiveness to MINERD shifting priorities, respondents asserted that PUCMM demonstrated ability to overcome setbacks (e.g., leveraging public￾private funds to offset budget shortfalls). PUCMM was further recognized for its strong financial management and cost-effective strategies. In addition to utilizing existing CETT materials, assessment instruments, and personnel to launch program activities, PUCMM leveraged funding to implement and replicate program components. For instance, PUCMM developed a partnership with the INICIA Foundation and MINERD in 2011 to expand the Math Teacher Training and School Management Components. MINERD printing and distribution of materials in ESP schools generated a savings for the program. ESP also successfully leveraged funds from INAFOCAM to expand the IE specialization program, and worked with a local organization in Ecuador to disseminate over 1,000 books for children with disabilities to ESP schools. Through building off of existing CETT materials, MINERD printing of Math textbooks, and leveraging public and private sector funds for program expansion, PUCMM efficiently managed the program. Though the bulk of the USAID investment was injected into the reading component, the investment produced high returns through improved student achievement and the development of well-trained teachers and leaders throughout the educational system. While teacher transfers pose a threat to the sustainability of the investment in ESP schools, if teachers remain in the lower grades, there may be some under-representation of efficiency gains. Key challenges. The primary challenge reported by school principals was resistance to change at the beginning of the program. Principals had learned through experience and training under the leadership specialization program that in the initial phase of any program, there is an adjustment period. MINERD and PUCMM school management experts claimed that it takes about two years to achieve a cultural change and mental shift from an intervention. This observation was consistent with our findings. Teachers that attended the specialization trainings or were with the program for three or more years admitted that the rigorous training was challenging at the beginning. However, once they began to experience results in the classroom, they realized the sacrifice was worthwhile and became convinced of the ESP methodology. Additionally, teachers found trainings on Saturdays difficult to attend due to personal obligations. Thus, schedule conflicts and resistance to change likely account for one-quarter of targeted teachers not participating in the in-service trainings. The second greatest challenge was teacher transfers. In every school visited during the evaluation, principals stated that many teachers trained under the ESP had transferred to other schools at the request of MINERD. One teacher enamored with the program requested to transfer to another ESP school. In many cases, there were only one or two ESP teachers remaining in ESP schools. Findings, Conclusions and Recommendations 25 Lack of coordination with MINERD in planning and implementation of the project due to high turnover and shifting government priorities was the third greatest challenge. There were many examples providing evidence of limited coordination. For instance, materials channeled through the Ministry were delayed or not received. Teachers received conflicting feedback from observations conducted by MINERD technicians and ESP mentors. There were schedule conflicts between trainings and events organized by MINERD and PUCMM. Frequent changes in MINERD and shifting education strategic priorities complicated coordination, but the situation is improving now that MINERD has witnessed the impact of the project and has adopted the Math and Reading components. Fidelity of implementation is another key challenge. Teachers and principals noted that developing teaching aids is time-consuming. Teachers spent two to three hours preparing materials for class. Many teachers requested mobile pre-printed posters and stories so they do not need to write all stories on flipchart paper. One principal stated that it takes planning time away from other subjects. Because it is labor-intensive to prepare materials and classroom displays, teachers in multiple shift schools feel discouraged from investing the time. Hence, multiple shifts pose a barrier to fully implementing reading corners and creating print-rich classrooms, which are important for promoting high reading achievement. With regards to the SS component, school principals informed that there was no formal requirement or mechanism for reporting cases of child abuse or violence. During the project, however, PUCMM maintained a record of every instance of child abuse identified, investigated the case, and worked with prosecutors to pursue redress. The evaluation team observed that the best results for the SS component were in schools in which the principal was trained in the SS component and worked with the community to address GBV cases. There were many challenges observed with the IE component (see Question 4). The most common challenges cited were: the delayed appointment of the IE Coordinator, the absence of IE materials, and the lack of follow-up support. Collaboration. The ESP had very limited coordination with other USAID projects due to different target populations, geographical areas7, or divergent objectives. There were a few attempts to train teachers and community members in the ESP reading methodology in schools under AMCHAM or Save the Children, but it was not very effective for the reasons cited above. The ESP and its partners learned that overlapping projects in the same school is not beneficial for three reasons. First, it affects fidelity to implementation. When there are multiple interventions, teachers and principals become confused about which intervention to prioritize and they lose sight of the goal. As a result, teachers do not fully implement the program as intended. Secondly, it is difficult to measure impact. It is not possible to attribute results to a specific intervention when there are multiple interventions. Thirdly, it complicates the reporting process. For instance, USAID runs the risk of double-counting beneficiaries who receive different interventions in the same school. Conclusions The key strengths, challenges and lessons learned of the ESP and final conclusions are outlined below. 7 The evaluation team found only five AMCHAM schools that overlapped with ESP: Ercilia Pepin, Rosa Duarte, Marcelino Valenzuela, Las Carmelitas and Juan Pablo Duarte. Findings, Conclusions and Recommendations 26 Findings Conclusions Key Strengths Finding #1: 50% of respondents lauded PUCMM for its excellent management, training, efficiency and sustained support. Finding #2: High qualified staff with extensive experience working with MINERD, USAID, and their components accelerated program results. Finding #3: PUCMM was recognized for its strong financial management and cost-effective strategies. ESP leveraged existing models, partnerships, and public￾private funds to enhance or sustain program components. Conclusion #1: PUCMM had excellent technical and financial management of activities across the components; the majority of beneficiaries were very pleased with the intensive and effective support received from PUCMM. Key Challenges Finding #4: Teachers' resistance to change was the greatest challenge reported by principals; school management experts claim it takes two to three years to create cultural and mental shifts. Finding #5: Scheduling conflicts likely prevented 25% of teachers from participating in trainings. Conclusion #2: Although it takes two to three years to create a cultural shift, now that the program has achieved results, it has the potential for rapid scale-up in ESP schools and districts. Finding #6: Teacher transfers were the second key challenge. Conclusion #3: Teacher transfers are frequent and rampant. Finding #7: The third most significant challenge was lack of coordination with MINERD in planning and implementation of the project due to high turnover and shifting government priorities. Conclusion #4: Throughout the project, there was lack of coordination with MINERD; yet the recent adoption of ESP methodology and coordination with PUCMM for scale-up shows promise for future collaboration. Finding #8: Development of teaching aids is time￾consuming. Finding #9: Multiple shifts prevent teachers from developing reading corners and print-rich classrooms, which are essential for promoting high reading achievement. Conclusion #5: Additional resources and infrastructure are necessary to achieve fidelity to implementation in overcrowded and low-resourced schools. Finding #10: For the SS component, best results were in schools in which the principal was trained and worked with the community to address cases; in most schools, there was no formal mechanism for reporting cases of child abuse. Conclusion #6: In many ESP schools, there was a lack of school-community and school-government partnerships and mechanisms to address cases of child abuse. Finding #11: For the IE component, the delayed appointment of an IE Coordinator, lack of materials and insufficient follow-up support reduced the component’s impact. Conclusion #7: Appointment of IE Coordinators/special education teachers is critical for fully implementing the IE strategy. Lessons Learned from Collaboration Efforts Finding #12: Collaboration affects fidelity to implementation and minimizes impact due to competing priorities. Finding #13: When collaborating it is difficult to measure impact and attribute results to a specific intervention or program. Conclusion #8: Overlapping projects in the same schools is not recommended as it creates conflict, minimizes the impact of the intervention, and complicates measurement and reporting of outcomes. Recommendations Recommendation #1: Continue supporting the integration of ESP into MINERD to sustain and replicate the program’s success. Findings, Conclusions and Recommendations 27 Based on: Conclusion #1: PUCMM had excellent technical and financial management and coordination of activities across the components. The majority of beneficiaries were very pleased with the intensive and effective support received from PUCMM. Recommendation #2: Extend the program in ESP schools with high turnover, offer training during the convenient times and locations to increase participation, and build the capacity of trained teachers and district technicians to provide ongoing mentoring. Based on: Conclusion #2: Although it takes two to three years to create a cultural shift, now that the program has achieved results, it has the potential for rapid scale-up in current ESP schools or in schools where ESP-trained teachers have transferred. Recommendation #3: Work with MINERD to sustain trained teachers in the lower grades. Based on: Conclusion #3: Teacher transfers are frequent and rampant. Recommendation #4: Develop a coordinated strategy for USAID, MINERD, and the implementing partner to collaborate on project design, implementation and evaluation. Recommendation #5: Involve the MOE in the planning the chronogram of activities to avoid schedule conflicts and in order to communicate a unified message to schools. Based on: Conclusion #4: Throughout the project, there was lack of coordination with MINERD; yet the recent adoption of ESP methodology and coordination with PUCMM for scale-up shows promise for future project collaboration. Recommendation #6: Provide pre-printed stories per grade, manipulatives and laminated cards or other materials that can be recycled to reduce prep time; another cost-effective option is to incorporate materials development into training workshops. Recommendation #7: Dedicate a classroom for safekeeping of reading materials and full implementation of the program (e.g., reading corners, classroom displays); some schools have dedicated the pre-school classroom as the literacy classroom since it is not used for multiple shifts. Based on: Conclusion #5: Additional resources and infrastructure are necessary to achieve fidelity to implementation in overcrowded and low-resourced schools. Recommendation #8: Establish school-community networks for systematic reporting of cases of child abuse or GBV; work with government to strengthen case management and law enforcement. Based on: Conclusion #6: In many ESP schools, there was a lack of school-community and school-government partnerships to address cases of child abuse. Recommendation #9: Work with MINERD to appoint and train IE Coordinators who can support students with special needs and assist teachers with differentiating activities in the classroom. Based on: Conclusion #7: Appointment of IE Coordinators/special education teachers is critical for implementing the IE strategy. Recommendation #10: One implementer should be in charge of all interventions in a school to eliminate conflict and maintain fidelity to implementation. Based on: Conclusion #8: Overlapping projects in the same schools is not recommended as it creates conflict, minimizes the impact of the intervention, and complicates measurement and reporting outcomes. Evaluation Question 6: Sustainability: Contributions to DR Education System Q6: WHAT ARE THE PERCEIVED CONTRIBUTIONS OF THE ESP TO THE DR PUBLIC PRIMARY SCHOOL SYSTEM? Findings, Conclusions and Recommendations 28 Findings All respondents were asked to identify contributions to the education system. The most significant contributions of the program to the Dominican Republic primary school system are: a revised primary school curriculum for the first cycle, improved student achievement in Math and Reading Comprehension, the development of classroom libraries and children’s books, a refined professional development model, the creation of specialization programs, improved school leadership, and the integration of students with disabilities into school activities. The most noteworthy achievement of the ESP, according to MINERD key informants, was curriculum revision for Math and Spanish Language. ESP supported MINERD with redesigning the curriculum for Math and Spanish Language in the first four grades of primary school. MINERD informed that they have adopted the teachers’ guide for grades 1-4 in Math and Spanish; the Spanish Language workbook for grades 1-3; the integrated workshop strategies guide in grades 1-4; and Math kits. In the northern region, the Ministry is now starting to implement PEF-MINERD – replicating the effective reading methodology in non-ESP schools with support from and in collaboration with PUCMM. Based on interviews with key informants, the program has achieved buy-in at all levels of MINERD. Although the findings reflect a small sample of MINERD representatives, results are promising. A national MINERD director noted that the ESP had the highest gains of three MINERD programs per an Inter-American Development Bank (IDB) study. A school principal stated that the ESP is the best program he has seen “in over 20 years in the education system.” Regional directors reported significant differences in the reading levels of students in ESP schools compared to non-ESP schools. One regional director who attended a reading fair claimed that a student in an ESP-school could read more fluently than most adults. The most significant accomplishment of the ESP according to school-level stakeholders was improved student achievement in Reading and Math. There has been observed improvement in students’ writing and reading comprehension skills in all grades (1-4). Principals have also observed improvements in students’ oral expression, critical thinking and creativity skills. Use of diverse materials, production of student’s original stories, and the interactive and dynamic approach in the classroom contributed to these outcomes. Peer mentoring was another effective approach. In one school visited, “reading promoters” in second grade mentored fourth graders in reading. Classroom libraries also enhanced students’ motivation to read and write. It was so effective in producing results that MINERD and private partners have provided additional library books for all grades in some schools. PUCMM also rewarded high-achieving schools with additional library books through a $100,000 innovation grant received in 2010 from the United Nations Educational, Scientific and Cultural Organization (UNESCO). PUCMM has further developed a reading and writing center within the university. The center reviews books for MINERD and recommends which are grade￾appropriate. Another noteworthy accomplishment of the program was the publishing of children’s books and development of children authors, which has motivated students, teachers and parents to engage in reading activities. ESP printed the books each year of the program. For 2015, the Ministry printed the books and brought them to the ESP closing ceremony as a gesture of appreciation. ESP refined the CETT professional development model and built a cadre of specialized literacy teachers, high quality mentors and dynamic classrooms for replication. Many teachers who thought they knew how to teach literacy well claimed that through ESP they felt like they were learning how to be a teacher for the first time; they learned so many new strategies about the phases of reading skills Findings, Conclusions and Recommendations 29 acquisition that the capacity building empowered them to be effective teachers in the classroom. As one teacher exclaimed, “I felt like I was sleeping for 20 years; now, I am a trained teacher because of PUCMM!” Teachers also shared that at first there was a lot of resistance to the program because of the time commitment and dedication required. Although it was time-consuming and difficult to adjust their methods, they claim it paid off. Teachers now incorporate four competencies a day into their lessons. The intensive and personalized mentoring tremendously contributed to improving the quality of instruction and achieving results. Teachers and mentors developed a personal relationship in many schools and continue to inform each other of progress and challenges. A sustainable contribution of the program was the creation of specialization programs. ESP created specialized programs in IE, Reading and Leadership. The program provided over 600 scholarships for advanced certification programs, with over 300 for IE specifically. INAFOCAM has decided to continue financing the IE program in Santiago and develop a similar program in Santo Domingo. ESP improved school management and leadership of over 2,000 principals in response to a request from the Ministry. ESP conducted the leadership training using the Ministry’s “School Principals” training module. Additionally, 391 principals received systematic training and mentoring on school management and leadership. Another 107 obtained specialization degrees. Through IE and SS activities, ESP improved students’ self-esteem, behavior, and communication skills. These results contributed to integrating students with special needs into classroom activities and a reduction in bullying. As a result of IE activities, more teachers have increased awareness of how to support students with special needs, and are developing differentiated plans to address diverse abilities in the classroom. However, they still require additional assistance with diagnosis, treatment and differentiating tasks in the classroom. Conclusions A summary of findings and their respective conclusions are presented in the table below. Findings Conclusions Finding #1: ESP influenced the re-design of the MINERD primary education curriculum. Finding #2: ESP contributed to improved student achievement in Math and Reading, with regards to students' writing, comprehension, oral language and critical thinking skills. Conclusion #1: ESP contributed to strengthening reading and math curriculum design, instruction and student achievement. Finding #3: ESP published children's book and developed child authors, which motivated the school community to engage in reading activities. Finding #4: ESP refined the CETT professional development model and built a cadre of specialized literacy teachers, high quality mentors and dynamic classrooms for replication. Finding #5: ESP created specialized certification programs in IE, Reading and Leadership. Finding #6: ESP improved school management and leadership skills of over 2,000 principals. Conclusion #2: PUCMM is working with MINERD to integrate and scale-up the program, but more support is needed with the transition. Finding #7: IE and SS activities improved students’ self esteem and communication skills, which contributed to greater acceptance of special needs students and less bullying. Finding #8: Teachers have increased awareness of Conclusion #3: There was insufficient time and scale for the SS and IE components to create systemic change, but there are signs of improvement. Final Thoughts and Conclusions 30 Findings Conclusions how to support students with special needs, but still require additional assistance with treatment and diagnosis. Recommendations Recommendation #1: Build the capacity of MINERD technical staff to fully integrate and sustain ESP-MINERD (e.g., train technicians in the classroom observation tools, promote retention of trained teachers in the lower grades, continue training teacher coordinators to mentor new and existing teachers, institutionalize teacher innovation circles). Recommendation #2: Develop a mechanism for sustaining the reading competitions and book publishing. Recommendation #3: Replicate the program; provide schools and teacher-training institutes with videos of training sessions and dynamic classrooms and training manuals so they can replicate the approach with new teachers. Recommendation #4: Work closely with ISFODOSU to continue providing specialized degrees and with INAFOCAM to fund scholarships. Based on: Conclusion #2: PUCMM is working with MINERD to integrate and scale-up the program, but more support is needed with the transition. Recommendation #5: Integrate all components from the inception of the project under one results framework. Based on: Conclusion #3: There was insufficient time and scale for the SS and IE components to create systemic change. FINAL THOUGHTS AND CONCLUSIONS The ESP significantly contributed to the MINERD primary education system. It has developed an effective professional development program and built a cadre of specialized reading teachers, coaches, and models for replication. There are promising signs of sustainability with the MINERD adoption of the Reading and Math components. The Ministry should continue disseminating teaching and learning materials for grades 1-4 and providing support with implementation. The future USAID program should concentrate on sustaining the reading program in grades 1-3, strengthening the capacity of teachers in fourth grade, and expanding the program through at least sixth grade in current ESP schools. The future program should also consider working closely with teacher coordinators and model teachers to replicate the Reading component in ESP schools and districts. The IE and SS components require further sensitization, concentrated resources, and a well-conceptualized strategy to increase their effectiveness. Given the lack of human and financial resources within the country to provide an IE specialist for each school, USAID should consider developing model IE schools that serve as resource centers. For the SS component, the program should emphasize long-term national strategies and interventions to achieve cultural change across schools and communities. For example, the program could work with MINERD to develop a national vision of a safe school and implement policies to enforce child protection and student codes of conduct, involving all levels of MINERD (e.g., PTA). Finally, the program components should be well-integrated and linked to one development hypothesis. The goal should be broadly communicated to all stakeholders and integrated into all trainings and sensitization activities. Annex A. Evaluation Statement of Work 31 ANNEX A. EVALUATION STATEMENT OF WORK I. OBJECTIVE OF THE EVALUATION The objective of the Effective Schools Program (ESP) final performance evaluation is twofold: 1. To assess the effectiveness of the USAID/Dominican Republic (USAID/DR) ESP in achieving its performance goals and its contributions to the Dominican primary education system; and 2. To inform implementation of ongoing USAID/DR work supporting improved primary education with a focus on reading. II. BACKGROUND – EFFECTIVE SCHOOLS PROGRAM (ESP) USAID/DR began its emphasis on early grade reading in the Dominican Republic in 2002, with its participation in the USAID Centers for Excellence in Teacher Training (CETT) program. CETT was a USAID regional program functioning in three regions and 14 countries in Latin America and the Caribbean which ended in 2010. The program was developed to improve literacy from first through third grades, and was implemented in the Dominican Republic by the Pontificia Universidad Católica Madre y Maestra (PUCMM). In 2006, USAID/DR added an additional program with PUCMM called the “Activities to Improve the Effectiveness of Teachers and Local Governance” project, commonly referred to as the TEF project. TEF was designed to improve learning in mathematics in the first cycle of primary education (1st – 4th grades) and to complement the CETT program by including a literacy component for the fourth grade. A governance (school management) component was integrated as part of this program in the second year of implementation. The TEF program ended on August 31, 2009. In June 2009, PUCMM submitted an application to USAID for a five-year follow-on program, called the Effective Schools Program (ESP), which USAID approved for support in October 2009. The ESP program takes an integrated and comprehensive approach to impact the formal education system as a whole. The goal is for participating schools to be changed into learning communities aimed at improving the quality of education in the Dominican Republic. The components of the program are: (1) Education/School Management; (2) In-Service Teacher Training in Literacy and Mathematics and Curriculum Development; and (3) Monitoring and Evaluation. ESP project components emphasize measurement of results through student achievement testing; teacher training and classroom support (including observation and assessment); and training and support for school administrators and school management structures that include parents. The original ESP end date of September 2014 was extended to April 2015. Two additional components were added to ESP in 2013. The first was the “Safe Schools” activity. In the 2013-2014 school year, Safe Schools developed and delivered trainings for teachers, students, parents, and other members of the educational community in 91 pilot schools, so that they would jointly learn to define fair processes in dealing with school-related gender-based violence and put these processes into action to create a safe learning environment. The second additional ESP component was called “Inclusive Learning Opportunities for Children with Disabilities”. The overarching objective of the component was to work in 100 ESP schools “to build the capacity of public schools in the Dominican Republic to detect, diagnose, treat, and support school-aged children with disabilities”. One of the main objectives of the component was to create a model, jointly with Ministry of Education (MOE) staff, to identify disabilities in children attending public schools in the Dominican Republic and to make adequate referrals for professional diagnosis and treatment when possible. Activities included training of professionals, teachers, school personnel and community members to support children with special educational needs, as well as design of inclusive strategies to give them better learning opportunities. ESP was tasked with documenting strengths and weaknesses of this inclusive model of education through a comprehensive monitoring and evaluation program. It will be extremely important for the evaluation team to study Safe Annex A. Evaluation Statement of Work 32 Schools and inclusive education activities under ESP, as these were not yet underway when the mid￾term evaluation took place. Recommendations as to how to improve the effectiveness of both Safe Schools and inclusive education programming will be extremely useful for shaping future USAID/DR education support, as USAID plans to continue work in both of these areas under newly planned education activities. II.A ESP Mid-Term Evaluation The ESP was evaluated in 2013 as part of an overall USAID/DR education portfolio mid-term evaluation. The team selected for this final performance evaluation must familiarize itself with the detailed information contained in that report, particularly the findings and recommendations specific to ESP.8 In addition to addressing the specific key questions outlined in Section III below, the evaluation team shall determine whether and how the following was addressed from Recommendation #6 of the ESP mid￾term evaluation: “Ensure student evaluations and teacher evaluations are implemented, analyzed and results disseminated to all levels of the education system to inform decision making.” II.B ESP Work as Part of a Ministry of Education Alliance In 2011, the Dominican Ministry of Education began a concerted effort to support early grade reading and mathematics through its “Policy to Improve Student Learning in the First Cycle of Primary School in Reading, Writing, and Mathematics.”9 Using InterAmerican Development Bank (IDB) funding and technical assistance combined with national funds, in order to implement the early grade policy, the MOE created an “alliance” of three institutions that have a long history of support for education in the Dominican Republic: the Organization of IberoAmerican States (OEI), the Dominican NGO “Centro Cultural Poveda,” and PUCMM. The MOE provided funding to each of the three partners to cover a specific geographic area of the country to support early grade reading and math for a period of four years in a defined set of schools with 500 students or more. The three alliance institutions began implementation of the early grade support policy with IDB funding in the 2012-2013 school year, each using their own selected pedagogical approaches, methodologies, and materials. PUCMM is working in the northern region of the country, implementing the ESP methodology and using ESP materials that have been developed over the years with USAID support. While only one full year of implementation had taken place, an IDB study in 2013 highlighted the Effective Schools Program’s learning efficiency and effectiveness vis-à-vis other institutional approaches.10 The MOE-supported alliance holds great potential for increasing sustainability of primary education activities in the Dominican Republic. The IDB’s findings and recommendations about the strength of the ESP approach can serve as a starting point for assisting the MOE to develop an integrated approach and standards that can be used by all implementing organizations working to support primary education improvement. The team for the ESP final performance evaluation is tasked with analyzing ESP activities as part of this MOE-supported alliance, recording any lessons learned and recommending how the ESP 8 USAID/Dominican Republic Education Portfolio Mid-term Performance Evaluation, January 2013, USAID, Amex International and DevTech Systems, Inc., http://pdf.usaid.gov/pdf_docs/pdacu985.pdf. Annexes at http://pdf.usaid.gov/pdf_docs/pdacx192.pdf. 9 USAID/DR will provide the evaluation team with the report “Politica de Mejoramiento de los Aprendizajes de los estudiantes del primer ciclo en Lectoescritura y Matematica” described in the document “Linea Base para la Sistematizacion, Componente de Apoyo a los Aprendizajes en Lectoescritura y Matematica,” Programa de apoyo al Plan Decenal de Educacion, MINERD-BID,” 2013. 10 Op. cit., page 38. Annex A. Evaluation Statement of Work 33 approach can assist the MOE to improve the quality of primary education support interventions nationwide. (The USAID/Dominican Republic’s overall results framework for FY 2008-2013 is attached to this document for reference. The ESP fell under the Mission’s basic education objective, i.e., Assistance Objective 3.) III. EVALUATION FOCUS AND KEY QUESTIONS The ESP final performance evaluation will follow guidance established in the USAID Evaluation Policy 2011.11 In line with that policy’s description of performance evaluations, it will focus on what ESP achieved at its conclusion; how it was implemented; how it was perceived and valued; whether expected results occurred; and other questions outlined below that are pertinent to program design, management and operational decision making. As noted earlier, the objectives of this evaluation are (1) to assess the effectiveness of the ESP in achieving its performance goals and its contributions to the Dominican primary education system and; (2) to inform implementation of ongoing USAID/DR work supporting improved primary education with a focus on reading. It should be stressed that this evaluation is not meant to reproduce the efforts of the mid-term evaluation. Rather, the final evaluation report shall draw upon key lessons learned to make recommendations that help to orient implementation of ongoing USAID/DR education programs. USAID/DR’s primary interest in following up the mid-term evaluation, and the top priority for this evaluation, is to better understand factors that affected reading achievement under ESP, as this will be the principal focus of the new education activity going forward.12 The following principal key research questions should guide the design, methodology, and outputs of the evaluation: 1. How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted communities? What factors are causing reading scores to remain low overall, even in cases where progress was made under the ESP? 2. How did the Safe Schools activity perform for boys and girls in terms of improving safety and creating an environment of tolerance in targeted schools? Were conditions for learning improved as a result of Safe School activities? Was there any noticeable correlation between a safe learning environment and academic improvement, as measured through improvements in reading? 3. How did the ESP perform in terms of meeting the requirements of targeted special needs boys and girls? Were changes in enrollment or academic performance observed as a result of ESP activities with special needs children? Evaluation findings must disaggregate data by gender and assess outcomes and impact on males and females for all evaluated activities. IV. GENERAL EVALUATION METHODOLOGY 11 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf. 12 USAID/DR will support a new basic education activity that incorporates many of the key focus areas implemented under ESP. The Request for Applications (RFA) for the new USAID/DR Lighting Excitement for Excellency in Reading (LEER) activity can be found at http://www.grants.gov/web/grants/search￾grants.html?keywords=dominican%20republic. Annex A. Evaluation Statement of Work 34 Planning and implementation of the evaluation study will be closely coordinated with the USAID/DR education team, the implementing partner PUCMM, and relevant MOE officials at national and local levels. The evaluators may draw upon the earlier mid-term evaluation and any analysis by others (e.g., IDB reports) in the final report in order to summarize the results achieved through ESP’s overall work. While USAID will work with the evaluation team to finalize a work plan and evaluation methodology for each of the three principal research questions posed above, an initial discussion of possible methodologies is presented below. In responding to question 1, the evaluators must utilize, but are not restricted to, the following methodologies: review ESP project documentation; conduct a detailed statistical analysis of the ESP student achievement database for reading; compare students in ESP and non-ESP schools and analyze any differences in achievement by sex, years of participation, and rural vs. urban location, as well as the problems related to over-age students; conduct key informant interviews with implementing project staff, beneficiary school directors, teachers, Ministry of Education staff at national and local levels, and beneficiaries to include students, parents, and involved community members. Group interviews and focus groups may be used when appropriate. In responding to question 2, the evaluators must utilize, but are not restricted to, the following methodologies: review ESP Safe Schools activity documentation; review curriculum and materials used for implementing the Safe Schools activity and analyze their relevance and usefulness in addressing the issues identified in participating schools; analyze the findings of Safe Schools survey instruments completed by students and teachers; analyze implementing partner monitoring and evaluation instruments and findings pertaining to changes in knowledge, attitudes, or behavior as a result of the Safe Schools program; conduct key informant interviews with implementing project staff, beneficiary school directors, teachers, Ministry of Education staff at national and local levels, and beneficiaries to include students, parents, and involved community members. Group interviews and focus groups may be used when appropriate. In responding to question 3, the evaluators must utilize, but are not restricted to, the following methodologies: review ESP project documentation related to inclusive learning opportunities; conduct an analysis of implementing partner tools developed to identify, assess, and/or and address the needs of special needs children and present findings; assess implementing partner training program for teachers, principals, and support personnel to assist special needs children; conduct key informant interviews with implementing project staff, beneficiary school directors, teachers, Ministry of Education staff at national and local levels, and beneficiaries to include students, parents, and involved community members. Group interviews and focus groups may be used when appropriate. V. TEAM REQUIREMENTS The evaluation team must have an appropriate mix of technical skills to conduct the evaluation. Experts should have a degree in Education, Economics, Development, Evaluation, or a Policy-related field. The contractor must be sensitive to local language capacity in the skills mix when proposing staff on the team. The evaluation team should make meaningful use of local evaluation experts whenever possible for analysis as well as data collection. Members of the evaluation team should be proficient in Spanish (to facilitate interviewing) and English (to facilitate writing the Evaluation Report and preparing any PowerPoint presentations). The following personnel are considered key to the evaluation team. The contractor may propose other personnel as needed to achieve the proposed evaluation approach: Annex A. Evaluation Statement of Work 35 1. Senior level OVERALL TEAM LEADER - this will be a full-time position for the duration of the evaluation. This person will serve as the primary point of contact between USAID and the evaluation team. The incumbent must have: • Bilingual English and Spanish language fluency. • A proven track record in terms of being highly qualified to lead, coordinate, and deliver evaluations. • Outstanding research skills and ability to synthesize large amounts of disparate information. • Excellent writing/organizational skills and proven ability to deliver a quality written product (Evaluation Report and PowerPoint); • Technical skills to manage the budget resources (dollars and staff) for the evaluation, as well as to assist and support the team with field logistics (e.g., coordinating with USAID and/or a government ministry to set up initial appointments for interviews). • Superior understanding of and prior experience (7+ years) with or on policy work (education and/or other sectors) in the Latin America and Caribbean (LAC) region. • A balanced, fair, unbiased, truthful approach to evaluation and reporting. 2. Senior-level BASIC EDUCATION SPECIALIST: • Strong, proven leadership and analytical skills. • Knowledge of education programs in the development context, evaluations, and general development issues. • Experience in implementing education programs in Latin American (preferable) and/or other challenging environments. • Experience working with programs proven to be sustainable after direct donor involvement ends. • Strong background and experience with education programs focused on improved student achievement in reading and mathematics. • Significant experience in evaluating donor-driven development programs. • A Master’s degree or greater level academic background in education or related field. 3. Mid-level PROGRAM ANALYST: • Proven ability to gather, analyze, and interpret data to articulate the benefit of education program interventions. • Proven ability to cull information from both qualitative and quantitative sources. • Significant experience in evaluating donor-driven development programs. • Strong relevant academic background. The contractor will need to consider, and budget accordingly, to what extent each of the Technical Specialists will require junior-level support (e.g., to assist in collecting, analyzing, and preparing tabular or graphic materials). At the same time, the overall efficiency with which the evaluation team can work likely will be enhanced by ensuring a junior-level administrative assistant who will work closely with the overall Team Leader to facilitate logistics for the evaluation team (e.g., helping to contact potential interviewees and set up interview appointments, especially if a potential interviewee needs to be interviewed at a distant location). The contractor may wish to consider proposing a staffing configuration that has a junior-level administrative assistant also providing support to one or more of the three Senior-level Technical Specialists. While the entire team will be responsible for all in-country logistical support, the overall Team Leader will have the primary responsibility as the Point of Contact between the team and the USAID mission. VI. DELIVERABLE Annex A. Evaluation Statement of Work 36 The consultant(s) undertaking the evaluation will conduct an initial briefing with members from the USAID/DR DO1 team, Program Office and Mission management to discuss the plan of action and once a week thereafter to brief USAID/DR on progress. A draft report with recommendations will be submitted to USAID/DR five (5) business days prior to the final briefing. The final report will be submitted six (6)13 weeks after the completion of the in-country assessment and should be no more than 30 pages in length, excluding annexes. The evaluation team will provide the following deliverables: 1. An evaluation report that contains, at a minimum, the following sections: a. Executive Summary b. Background and Key Research Questions c. Assessments d. Findings* e. Conclusions and Recommendations* f. References g. Annexes h. List of Tables * The Findings and Conclusions and Recommendations sections should be given priority level of effort, with no less than 15-20 pages devoted to them in the final evaluation report. Given that one of the evaluation’s primary objectives is to inform implementation of ongoing USAID/DR education work, the evaluation team shall identify, whenever possible, lessons learned, particularly from the ESP experience, and describe how these can inform new activities that USAID/DR is supporting in reading, Safe Schools, and inclusive education. Of particular interest to USAID/DR are observations relating to the following: 14 • What are the obstacles to coordination when there are multiple USAID projects (e.g., PUCMM’s ESP and AMCHAM’s BEP) operating in the same geographic area? • What are the advantages of multiple USAID projects operating in the same geographic area? Does USAID achieve greater development results when multiple projects operate in the same area? • Has the ESP contributed to reducing citizen’s perception of insecurity in targeted areas, in coordination with other USAID/DR programming focused on youth, crime prevention and institutional strengthening of the criminal justice system? The evaluation report shall include the Scope of Work as an annex. All modifications to the Scope of Work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, need to be agreed upon in writing by the Technical Officer. 13 This number is revised from the original specification of three weeks to six weeks. 14 The SOW’s request for observations and recommendations to inform implementation of new education activities might be described as more characteristic of a sector assessment than a performance evaluation. Nonetheless, “[t]here is no reason why a SOW can’t combine a sector assessment and an evaluation.” “Sector assessments tend to be more forward looking with an emphasis on what the needs are that should be addressed with our programming, while evaluations tend to be more backward looking regarding our efforts in addressing particular needs.” “One area where there are particular synergies in combining a sector assessment with an evaluation is in the development of recommendations. Combining information from a sector assessment and an evaluation can improve the recommendations made by an evaluator.” Taken from “Six Key Issues to Consider when Developing an Evaluation Statement of Work” by Jerome Gallagher, USAID/PPL/LER, 7/1/13. Annex A. Evaluation Statement of Work 37 The evaluation methodology shall be explained in detail. In addition, all tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides, will be included in an annex in the final report. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). The evaluation team will deliver three copies (two paper copies and one electronic copy in Word format) of the report to the USAID/DR DO1 Deputy Director. VII. PERFORMANCE REQUIREMENTS AND STANDARDS Result 1: Evaluation/Assessment Addressing Key Questions in Manner of Utility to USAID/DR Completed Requirement 1.1: Develop evaluation plan, including data collection and analysis plan and instruments. Standards: • Evaluation plan provided within 3 weeks.15 • Plan for evaluation includes data collection methodology, an analysis plan, and instruments to address the key questions listed in section III. • Plan identifies data sources that will be used for each question, including sources of data that are already available, such as monitoring reports and prior evaluations. • Plan includes consideration of how the prior USAID/DR education portfolio mid-term evaluation will be used to inform analysis and conclusions. • Plan does not duplicate the prior evaluation. • Evaluation plan and instruments are informed by discussion with USAID/DR. Data collection instruments, including any survey, its questions and recipient list, would be designed in close collaboration with USAID/DR. Requirement 1.2: Collect and analyze evaluation data. Standards: • Number of site visits to ESP project schools, including those where Safe Schools and inclusive education activities were undertaken, to be determined based on availability and agreement with USAID.16 • Key informants identified based on knowledge of the ESP program and a USAID-provided list of key informants. Close attention paid to the prior mid-term evaluation so as not to unnecessarily replicate data that is unlikely to yield new or relevant information to this evaluation. • Number of individuals surveyed or interviewed and/or focus groups held is sufficient as to provide meaningful representation and draw meaningful conclusions. This includes sufficient representation by each relevant stakeholder group and each activity. • Secondary data should be collected as it relates to the relevant ESP work to be evaluated. • Any conclusions developed are based on analysis of findings. 15 This number is revised from the original specification of two weeks to three weeks. 16 The number of site visits is revised from no less than 40 schools based on availability and agreement with USAID. Annex A. Evaluation Statement of Work 38 • Any recommendations are based on conclusions from analysis of findings, and/or other pertinent statistics and studies Requirement 1.3: Produce evaluation report. Standards: • USAID/DR input incorporated into report outline. • Draft report with recommendations submitted to USAID/DR five (5) business days prior to the final briefing. • Final report submitted six (6)17 weeks after the completion of the in-country assessment. • Final report incorporates USAID/DR input. • Evaluation report addresses all evaluation questions included in the Scope of Work. • Report includes executive summary, which summarizes significant points from the full report, including key findings and recommendations. Any information provided in the executive summary appears in the full report. • Report clearly distinguishes findings (facts), conclusions, and recommendations. • Evaluation findings assess outcomes and impact on males and females. • The logical connections between findings, conclusions, and recommendations are clear to the reader. Each conclusion is based on observations and specific findings, and each recommendation is clearly related to a conclusion. • Report clearly differentiates analysis, conclusions, recommendations, etc., related to the evaluation questions. • For any survey data, the report includes an annex presenting a detailed and organized summary of findings from the survey, including summary statistics and an overview of respondents. • Recommendations are action-oriented, practical and specific, with defined responsibility for the action. Requirement 1.4: Present findings of analysis. Standards: • Presentation includes all key findings. • Presentation includes a PowerPoint that summarizes findings. • Presentation is given to a diverse audience, with invitations to include USAID/DR and PUCMM. VIII. SUPPORTING INFORMATION VIII A. Place of Performance Performance of this evaluation will take place in the Dominican Republic. VIII B. Period of Performance The period of performance is a total of approximately 45 days (see Illustrative Timeline in Section IX below), beginning on/about March 15,18 2015, in Santo Domingo, Dominican Republic, in large part. 17 This number is revised from the original specification of three weeks to six weeks. 18 This date is revised from the original estimated start date of February 2 to March 15. Annex A. Evaluation Statement of Work 39 VIII C. Reporting Requirements and Logistical Arrangements The Deputy Director, Office of Youth, Education and Security, and/or her designee shall provide technical direction and general guidance during the performance of this task order. The consultants will report to the Deputy Director. VIII D. Copy of Report to Development Experience Clearinghouse In line with the USAID Evaluation Policy, completed evaluations must be submitted to the USAID’s Development Experience Clearinghouse (DEC) and a cover sheet attached indicating the type of evaluation conducted and the design. Each completed evaluation must include a 3- to 5-page summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation. IX. ILLUSTRATIVE TIMELINE (ASSUMES MONDAY TO FRIDAY AS BILLABLE WORKDAYS) The following provides a notional presentation of a prospective allocation of level of effort for the evaluation. The contractor is at liberty to develop the firm's own prospective allocation of level of effort (and accompanying budget). The final timeline is to be proposed by USAID/DR and agreed upon with the evaluation team. 5 Days (in contractor’s home office) – Review project documentation recommended by the Mission. The evaluation team will also identify any other relevant performance information sources, such as performance monitoring systems and/or previous evaluation reports, and any other type of data (quantitative or qualitative) available from implementers or other sources. This secondary data review will be used by the contractor to set forth the proposed methodologies to help identify major data gaps and data collection methodologies and to structure fieldwork. The USAID/DR evaluation manager will review the methodology and provide input prior to contractor undertaking fieldwork. The contractor will be responsible for refining the data collection; this may include coordinating and engaging with appropriate USAID personnel to discuss the design, plan and the objectives for measuring outputs and impacts. This may include data collection methodology and/or sampling to be employed, for example. 3 Days (in-country) – Initial orientation meetings with USAID, project implementers and/or relevant MOE staff. Finalization and approval of evaluation work plan. 15 Days (in-country) – In-depth interviewing of USAID staff and project implementers, partners, and beneficiaries to confirm project results, constraints to the project being more successful in achieving desired results, and stakeholder views on ways in which assistance could be more effective in achieving expected/desired results. 1 Day (in-country) – Mid-Term Briefing (approximately halfway into field work). 1 Day (in-country) – Final Briefing at end of field work. 7 Days (in contractor’s home office) – Prepare relevant summary tables, graphs, and annexes. Drafting of evaluation narrative, including Executive Summary and other content (tables, graphs, and annexes). 1 Day (in contractor’s home office) – Prepare a PowerPoint presentation. 1 Day (teleconference) – Deliver a Final Briefing (with PowerPoint) to brief USAID on the evaluation’s findings, conclusions, and recommendations; and to discuss Mission comments on the Evaluation. Annex A. Evaluation Statement of Work 40 2 Days (in contractor’s home office) – Wrap up draft and submission of draft Evaluation Report and Final Briefing (with PowerPoint). Mission will have 10 business days to review the draft Evaluation Report and provide written comments back to the Evaluation Team. 2 Days (at contractor’s home office) – Revise draft Evaluation Report and PowerPoint to address comments provided by USAID. Submit to USAID evaluation manager final Integrated Evaluation Report and PowerPoint presentation. The estimated period of performance of this task order will be 45 days, beginning on/about March 15,19 2015. Estimated Date: 20 Action / Deliverable: 3/12/15 Planning meeting(s) with evaluation team and USAID/DR 4/3/15 Draft evaluation design for review by USAID 4/3/15 Draft instruments for review by USAID 6/12/15 Draft evaluation report for review by USAID 6/18/15 Briefing on evaluation findings and recommendations to USAID 7/6/15 Final report X. KEY REFERENCES AND READING FOR EVALUATORS The following documents are mandatory background reading for the evaluation team and can either be found on-line or will be provided prior to initial meetings to discuss the evaluation design with USAID/DR: 2. USAID/Dominican Republic Education Portfolio Mid-term Performance Evaluation (including Annexes), January 2013. http://pdf.usaid.gov/pdf_docs/pdacu985.pdf 3. Annexes at http://pdf.usaid.gov/pdf_docs/pdacx192.pdf 4. Linea Base para la Sistematizacion, Componente de Apoyo a los Aprendizajes en Lectoescritura y Matematica, Programa de apoyo al Plan Decenal de Educacion, MINERD-BID, 2013 5. ESP Cooperative Agreement with PUCMM 6. USAID/DR Country Development Cooperation Strategy FY 2014-2018 7. http://www.usaid.gov/sites/default/files/documents/1862/Dominican%20Republic%20CDCS_publi c%20version_FY14_FY18.pdf 8. DR TDY Report and Gender Action Plans for DO Teams, Regina Jun, USAID/DR Mission document, December 2013 9. Request for Applications (RFA) for the USAID/DR Lighting Excitement for Excellency in Reading (LEER) http://www.grants.gov/web/grants/search-grants.html?keywords=dominican%20republic 10. USAID Evaluation Policy (2011) http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf 11. USAID Gender Equality and Female Empowerment Policy (2012) http://pdf.usaid.gov/pdf_docs/PDACT200.pdf 19 This date is revised from the original estimated start date of February 2 to March 15. 20 TBD dates are replaced with provisional dates. Annex A. Evaluation Statement of Work 41 USAID/Dominican Republic Goal FY 2008-2013 To help build and sustain democratic, well-governed states that responds to the needs of their people, reduce widespread poverty and conduct themselves responsibly in the international system Assistance Objective 1 Enhanced Accountability, Transparency, and Responsiveness to Citizens within Key, Selected Government Institutions Assistance Objective 2 To Generate Rapid, Sustained, and Broad￾based Economic Growth Assistance Objective 3 Quality of Basic Education Improved Assistance Objective 4 Improvement in the health of the most vulnerable populations in the Dominican Republic. Justice Sector performance improved to be more independent, transparent, efficient, and fair Core governance institutions performance improved to be more effective at ensuring transparency and accountability within the Executive Branch Improved organized citizen participation in the implementation of key policy reforms, generating demand for transparency, exercising government oversight Strengthened Public/Private Sector Capacity to Implement DR-CAFTA Improved market linkages of agriculture and wood products produced by MSMEs Increased sustainable tourism practices in selected tourism destinations Improved environmental protection in the D.R. Student Performance in 1st through 4th Grade Reading and Math Improved Community and Private Sector Involvement in Public Education Strengthened Learning Opportunities For At-Risk Youth Increased Increased use of services and adoption of practices to prevent and mitigate the spread of HIV/AIDS. Improved quality of services for the prevention, detection and treatment of infectious diseases, with emphasis on tuberculosis. Increased use of effective voluntary family planning (FP) services. Increased access to and delivery of quality maternal and child health care services in selected areas. Annex B. Evaluation Planning Matrix 42 ANNEX B. EVALUATION PLANNING MATRIX READING Evaluation Question Data Collection Method Data Collection Source Sample Size How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted communities? • Document Review • Analysis of Student Achievement Data • Key Informant Interviews • Quarterly/Annual Reports • Student Achievement Data • Baseline Survey • Results of EGRA/EGMA • CETT White Papers Series • Project Staff (COP/M&E) • MINERD Technical Staff • School Directors • Trained Literacy Teachers • 49 Schools • 19 National and District Key Informants What factors are causing reading scores to remain low overall, even in cases where progress was made under ESP? • Document Review • Key Informant Interviews • Classroom Observations • LEER RFP • ESP Project Documents • Student Achievement Tests • Project Staff • MINERD Technical Staff • School directors • Trained literacy teachers • Literacy class • 49 Schools • 19 National and District Key Informants Which reading approach (whole word or phonics) has the program adopted and why? Has the project tried a balanced approach? • Key Informant Interviews • Classroom Observations • Project Staff • MINERD Technical Staff • Trained Literacy Teachers • Literacy Class • 47 Classroom Observations and Interviews • 1 Interview with PUCMM Literacy Expert Have student and teacher evaluations been conducted in ESP schools? To whom were the results disseminated? Were they disseminated to all levels of the education system to inform decision-making? • Document Review • Key Informant Interviews • Project Reports • Project Staff • MINERD Technical Staff • School Directors • 53 Schools • 19 National and District Key Informants Annex B. Evaluation Planning Matrix 43 SAFE SCHOOLS Evaluation Question Data Collection Method Data Collection Source Sample Size How did the Safe Schools activity perform for boys and girls in terms of improving safety and creating an environment of tolerance in targeted schools? • Document Review • Key Informant Interviews • Focus Group Discussions • PLA Group Activities • Observation Checklist • Community Transect Walks • USAID Safe Schools Program, General Information • Safe Schools Program Final Report • Quarterly/Annual Reports • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community • Students • 25 Schools • 19 National and District Key Informants Were conditions improved for learning as a result of Safe School Activities? • Document Review • Key Informant Interviews • Focus Group Discussions • PLA Group Activities • Observation Checklist • Community Transect Walks • Safe Schools Doorway Training Manual • Quarterly/Annual Reports • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community Students • 25 Schools • 19 National and District Key Informants Was there any noticeable correlation between a safe learning environment and academic improvement, as measured through improvements in reading? • Document Review • Key Informant Interviews • Focus Group Discussions • Observation Checklist • Quarterly/Annual Reports • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community • 25 Schools • 19 National and District Key Informants Provide recommendations on how to improve the effectiveness of Safe Schools for shaping USAID/DR education support. • Document Review • Key Informant Interviews • Focus Group Discussions • PLA Group Activities • IBD Effectiveness Study • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community • 25 Schools • 19 National and District Key Informants Annex B. Evaluation Planning Matrix 44 INCLUSIVE EDUCATION Evaluation Question Data Collection Method Data Collection Source Sample Size How did the ESP perform in terms of meeting the requirements of targeted Special Needs boys and girls? Were changes in enrollment or academic performance observed as a result of ESP activities with special needs children? • Document Review • Key Informant Interviews • Focus Group Discussions • Information on USAID disability policy and activities • PUCMM document, 2013 (Programa Especialidad en Inclusion Educativo de Ninos y Adolescentes con Discapacidad) • MINERD document, 2013 (Informe Preliminar sobre Levantamiento de la Información de Estudiantes con Discapacidad en las Escuelas Eficaces, Informe Preliminar) • Quarterly/Annual Reports • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community Students • 22 Schools • 19 National and District Key Informants Provide recommendations on how to improve the effectiveness of Inclusive Education programming for shaping USAID/DR education support • Document Review • Key Informant Interviews • Focus Group Discussions • IBD Effectiveness Study • ESP Documentation on Strengths and Weaknesses of the Inclusive Model • MINERD Technical Staff • Project Staff • School Directors • Trained Teachers • Parents/Community Members • Students • 22 Schools • 19 National and District Key Informants Annex B. Evaluation Planning Matrix 45 IMPLEMENTATION EFFECTIVENESS Evaluation Question Data Collection Method Data Collection Source Sample Size Did the project meet its expected results and targets as described in the cooperative agreement? • Document Review • Key Informant Interviews • Cooperative Agreement • Project Monitoring Report • Midterm Evaluation • Quarterly/Annual Reports • Project COP/M&E • Document review • Interviews with PUCMM monitoring staff How well was the project implemented, coordinated and managed? What were the challenges and lessons learned? • Document Review • Key Informant Interviews (See USAID coordination questions under priorities.) • Quarterly/Annual Reports • Midterm Evaluation • ESP Documentation on Challenges and Lessons Learned, Particularly on the Inclusive Model, Literacy or Safe Schools Activities, and Coordination with Other USAID Projects/Activities • MINERD Staff • Program Staff • Program Partners (AMCHAM) • School Principals • 53 Schools • 19 National and District Key Informants What are the perceived contributions to the DR public primary school system? (Successes/Accomplishments) • Document Review • Key Informant Interviews • Program Staff • MINERD Technical Staff • School Directors • Trained Teachers • Parents/Community • 53 Schools • 19 National and District Key Informants Annex C. Evaluation Methods and Limitations 46 ANNEX C. EVALUATION METHODS AND LIMITATIONS This section describes the instrument development process and data collection methods. Instrument Development: During the preparation phase, the team identified target groups for key informant interviews and focus group discussions and developed a semi-structured core questionnaire with key questions corresponding to each USAID evaluation question, which allowed for a level of standardization across interviews. Using the core questionnaire, the team adapted the instrument to suit each target group at the national, district, and school levels. The questionnaires were finalized after piloting the instrument with a school in La Vega. A classroom observation guide was also developed based on the team’s experience in past USAID evaluations and the 5Ts of effective literacy instruction. 21 All instruments are available in Annex D. Document Review: USAID/DR and PUCMM provided available monitoring data and reports for the ESP. These documents included program descriptions, Performance Management Plans, Quarterly and Annual Reports, training materials, presentations, and external evaluations or assessments of projects or project components. In some cases, requested documents were not available. Key Informant Interviews: The evaluation team conducted 168 semi-structured key informant interviews with direct and indirect program beneficiaries at the national, district and school levels. Informants consisted of USAID, MINERD and PUCMM representatives at the national level; ESP mentors and District/Regional Directors at the district level; and, teachers, principals, coordinators, counselors, and psychologists at the school level involved in the ESP program. Standard data collection instruments presented in Annex D were developed for each target group to collect both quantitative and qualitative data related to the key questions. Focus Groups: Focus groups with parents were conducted in 26 schools to learn about parents’ opinion of the ESP performance, remaining challenges, and recommendations for future programming. In schools with a Safe Schools component, the evaluation team attempted to organize youth focus groups. Youth focus groups were held in 18 schools with 171 youth. Youth were asked to describe the past and present situation of their school as it pertains to gender-based violence and to describe their ideal future school. Their perceptions of bullying, violence and safety were triangulated with principal and teacher responses to Safe Schools questions. Classrooom Observations: The team collected data on teachers’ reading instructional methods and students ability to read fluently with comprehension through classroom observations in first through fourth grade. The criteria for selection was an ESP-trained teacher, and in many cases there was only one or two left at the school due to teacher transfers. In some cases, the principal directed the team to the teacher who was still using the ESP methodology, had received the most training and mentoring, and who had adopted the approach versus a teacher who was less comfortable or not as well trained. In 21 Consistent with USAID performance measures, the evaluators assessed “5Ts” of effective language instruction: (1) tongue – teaching in the local language, (2) text - ensuring the pupils are reading from printed text, (3) time on task (30 minutes for reading and writing), (4) testing of reading skills, and (5) use of appropriate teaching methods (teaching phonics/comprehension, modeling correct pronunciation, etc.). Annex C. Evaluation Methods and Limitations 47 these few instances, principals explained that ESP concentrated mentoring in the lower grades (1-3) and not fourth grade. The team viewed a representative sample of teachers in each grade. Classroom observations took place in 47 schools with 23% taking place in first grade, 28% in second grade, 19% in third grade and 28% in fourth grade. SAMPLING METHODOLOGY The schools were selected using a purposeful sampling methodology. The team obtained a list of schools organized by component and a list of schools ranked by reading achievement scores. Using the database, schools were selected per the following criteria: (1) schools with all five components; (2) schools with the highest and lowest reading achievement scores on fourth grade tests administered in 2012 and 2014; (3) extended day schools; and, (4) schools with safe school or inclusive education only interventions. The table below shows the corresponding breakdown of the schools by component and criteria and percentage of target population represented in the sample. Table 7. School Selection Criteria Criteria Total Evaluation Sample Total ESP Schools % of Total 5-Component School 11 15 73% High-Reading Achievement* 14 12 117% Low-Reading Achievement* 15 10 150% Total Reading 49 400 12% Safe Schools 25 91 27% Inclusive Education 22 55 40% Extended Day School 6 25 24% Total Schools 53 400 13% Source: Fieldwork data. *For the high and low-reading achievement school selection, schools ranking in the bottom 12 or in the top 10 on ESP reading comprehension tests were sampled. The ranges of scores are shown below. Schools in the bottom range scored between 31-39% on third and fourth grade tests in 2012 and 2014; those in the top range scored between 51.9% and 69.8%. Table 8. Reading Comprehension Scores Year Evaluated Lowest Scores Highest Scores Average 2012 31.1% - 36% 51.9% - 58.5% 41.4% 2014 30.8% - 39.4% 57.9% - 69.8% 48.3% Source: ESP school achievement data. Annex D. Data Collection Instruments 48 ANNEX D. DATA COLLECTION INSTRUMENTS This Annex contains the following evaluation instruments: 1. USAID Stakeholder Questionnaire; 2. National Stakeholder Questionnaire; 3. District Stakeholder Questionnaire; 4. School Principal Questionnaire; 5. Spanish Language Teacher Questionnaire; 6. Spanish Language Classroom Observation Tool; 7. Safe Schools School Staff Questionnaire; 8. Inclusive Education School Staff Questionnaire; 9. Parents Focus Group Tool; and 10. Youth Focus Group Tool. Annex D. Data Collection Instruments 49 USAID STAKEHOLDER QUESTIONNAIRE Name: ___________________________ Position: _________________________ Date: _____________________________ Time: ___________________________ Interviewer: _______________________ Location: _________________________ Introduction: Thank you very much for your participation in the evaluation. The purpose of the evaluation is to learn about your experience in relation to the Effective Schools program, your opinion about the key accomplishments and lessons learned, and the contributions to the Dominican public primary school system. In particular, we are interested in learning about the results of the Spanish language, safe schools and inclusive education components. You will be asked to make recommendations to improve the ESP program and future USAID/DR programs. Your responses will be confidential. Introductory Questions 1. How long have you been in your current position? ________________ 2. How long have you been working with the project? _______________ 3. In which project components and activities have you participated? ! In-service training workshop (Literacy) ! In-service training workshop (Math) ! Management training ! Safe Schools training ! Inclusive education trainings ! All five components ! Other ___________________________________________ Key Questions 1. Has the project achieved its expected results and targets per the contract and work plan? If yes, how do you track results? (Request copy of results achieved to data against targets from COP and/or M&E specialist.) 2. In your opinion, what are the most significant accomplishments of the Effective Schools Project (ESP)? What has it contributed to the DR public primary school system? 3. What have been the most significant challenges or lessons learned? Effectiveness 4. How well was the project implemented, coordinated and managed? What were the strengths? What were the challenges and lessons learned? USAID Coordination (Questions for PUCMM and USAID only) 5. Does ESP overlap with other USAID projects? If yes, which ones? Annex D. Data Collection Instruments 50 6. Does ESP coordinate with these projects? If yes, how? 7. What are the advantages of multiple USAID projects operating in the same geographic area? What are the challenges or obstacles to coordination where there are multiple USAID projects (e.g., PUCMM’s ESP and AMCHAM’s BEP) operating in the same area? 8. Does USAID achieve greater impact when projects operate in the same area? (If yes, why? If not, why not?) 9. Has the ESP contributed to reducing citizen’s perception of insecurity in targeted areas, in coordination with other USAID/DR programming focused on youth, crime, prevention and institutional strengthening of the criminal justice system? Final Recommendations 10. If replicated, what recommendations do you have for improving the effectiveness of the USAID/DR ESP program? Probe for each component below: ! Recommendations for Reading component ! Recommendations for Safe Schools component ! Recommendations for Inclusive Education component ! Other general recommendations (project implementation) Annex D. Data Collection Instruments 51 NATIONAL STAKEHOLDER QUESTIONNAIRE Name: ___________________________ Position: _________________________ Date: _____________________________ Time: ___________________________ Interviewer: _______________________ Location: _________________________ Introduction: Thank you very much for your participation in the evaluation. The purpose of the evaluation is to learn about your experience in relation to the Effective Schools program, your opinion about the key accomplishments and lessons learned, and the contributions to the Dominican public primary school system. In particular, we are interested in learning about the results of the Spanish language, safe schools and inclusive education components. You will be asked to make recommendations to improve the ESP program and future USAID/DR programs. Your responses will be confidential. Introductory Questions 1. How long have you been in your current position? ________________ 2. How long have you been working with the project? _______________ 3. In which project components and activities have you participated? ! In-service training workshop (Literacy) ! In-service training workshop (Math) ! Management training ! Safe Schools training ! Inclusive education trainings ! All five components ! Other ___________________________________________ Key Questions 1. Has the project achieved its expected results and targets per the contract and work plan? If yes, how do you track results? (Request copy of results achieved to data against targets from COP and/or M&E specialist.) 2. In your opinion, what are the most significant accomplishments of the Effective Schools Project (ESP)? What has it contributed to the DR public primary school system? 3. What have been the most significant challenges or lessons learned? 4. Did the project conduct student and teacher performance evaluations in ESP schools? If yes, did you receive information on the outcomes? ! Yes, received information on results of evaluations ! No, did not receive information on results of evaluations 5. Which stakeholders within the Ministry of Education received the results of the teacher and student performance evaluations, and at which levels of the education system? ! School level _______________________ ! District level _______________________ Annex D. Data Collection Instruments 52 ! National level ______________________ Reading 6. How did the ESP perform in terms of improving reading skills of primary school boys and girls in targeted schools? On average, did reading comprehension scores increase during the time period (2009-2015) of the project? ____Yes, for girls; ____Yes, for boys: _____Yes, for both ____Not for girls; _____ Not for boys; _____No, for both 7. What factors have you found contribute to low reading achievement in ESP schools that have received the interventions? 8. Which reading approach (whole language, phonics, balanced, or other) has the program adopted? Do you agree with this approach? Why or why not? Safe Schools 9. How did the Safe Schools activity perform in terms of improving safety for boys and girls and creating an environment of tolerance in targeted schools? (Probe: Is the school safer now for boys and girls? Did the learning environment improve? If yes, how so?; If not, why not?) ____ Yes, for girls; Explain ___________________ ____ Yes, for boys; Explain __________________ ____ Yes, for both; Explain ___________________ ____ Not for girls; Explain ____________________ ____ Not for boys; Explain ____________________ ____ No improvement for either boys or girls; Explain risks and challenges _____ 10. Are children using less abusive language? ! Yes ! No 11. Is there a reduction in violent behavior? ! Yes ! No 12. Has there been an improvement in attendance and/or academic performance as a result of a safe and improved learning environment? ! Yes (Explain: ________________________________________ ! No (Explain: _________________________________________ 13. Has a safe and improved learning environment contributed to an improvement in reading performance? ! Yes (Explain: ________________________________________ ! No (Explain: _________________________________________ ! Don’t know Special Needs 14. How did the ESP address the needs of boys and girls with disabilities? Annex D. Data Collection Instruments 53 15. Were the activities sufficient to address all the learning needs of children with disabilities? Why or why not? ! Yes (If yes, what was the result of the activities?) _____________ ! No, (If no, what else is needed?) ________________________ 16. Has enrollment of children with disabilities increased as a result of ESP activities? (Collect information on enrollment numbers from PUCMM) ! Yes, If yes, how much has it increased? # ____________ % _________ ! No 17. Has the academic achievement of students with disabilities improved as a result of the ESP activities? ! Yes; How much has performance improved on learning assessments? __ ! No Effectiveness 18. How well was the project implemented, coordinated and managed? What were the strengths? What were the challenges and lessons learned? USAID Coordination (Questions for PUCMM and USAID only) 19. Does ESP overlap with other USAID projects? If yes, which ones? 20. Does ESP coordinate with these projects? If yes, how? 21. What are the advantages of multiple USAID projects operating in the same geographic area? What are the challenges or obstacles to coordination where there are multiple USAID projects (e.g., PUCMM’s ESP and AMCHAM’s BEP) operating in the same area? 22. Does USAID achieve greater impact when projects operate in the same area? (If yes, why? If not, why not?) 23. Has the ESP contributed to reducing citizen’s perception of insecurity in targeted areas, in coordination with other USAID/DR programming focused on youth, crime, prevention and institutional strengthening of the criminal justice system? Final Recommendations 24. If replicated, what recommendations do you have for improving the effectiveness of the USAID/DR ESP program? Probe for each component below: ! Recommendations for Reading component ! Recommendations for Safe Schools component ! Recommendations for Inclusive Education component ! Other general recommendations (project implementation) Note to the evaluator: Complete the checklist for each interview. 1. Has the program achieved its expected results and ! Yes ! No Annex D. Data Collection Instruments 54 targets? 1c. Were the results of teacher and student performance disseminated to key stakeholder(s) at the national level? ! Yes ! No 3. Did the ESP improve reading skills of primary school boys and girls in targeted communities? ! Yes, girls ! Yes, boys ! Yes, both ! Not girls ! Not boys ! No, not either 4a. Did the learning environment improve as a result of Safe Schools activities? 4e. Did this have any impact on reading performance? ! Yes ! Yes ! No ! No 5b. Were ESP special needs activities sufficient to address all the learning needs of children with disabilities? ! Yes ! No 5c. Has enrollment of children with disabilities increased? Amount (#) ___________________ Amount (%) ___________________ ! Yes ! No 5d. Has academic achievement of children with disabilities increased? ! Yes ! No Annex D. Data Collection Instruments 55 DISTRICT STAKEHOLDER QUESTIONNAIRE School Name: District: Date: Evaluator: Name of Person Interviewed: Title: Introduction: Thank you very much for your participation in the evaluation. The purpose of the evaluation is to learn about your experience in relation to the Effective Schools program, your opinion about the key accomplishments and lessons learned, and the contributions to the Dominican public primary school system. In particular, we are interested in learning about the results of the Spanish language, safe schools and inclusive education components. You will be asked to make recommendations to improve the ESP program and future USAID/DR programs. Your responses will be confidential. Introductory Questions 1. How long have you been in your current position? ________________ 2. What is your role in the project? _____________ 3. How long have you been working with the project? _______________ 4. In which project components and activities have you participated? ! In-service training workshop (Literacy) ! In-service training workshop (Math) ! Management training ! Safe Schools training ! Inclusive education trainings ! All five components ! Other ___________________________________________ 5. What was your opinion of the training received? Was it sufficient for you to effectively carry out your role? Key Questions 1. What is your opinion of the overall performance of the ESP project? Has the project achieved its expected results per the work plan? If yes, how do you measure results? Annex D. Data Collection Instruments 56 2. In your opinion, what are the most significant accomplishments of the Effective Schools Project (ESP)? What has it contributed to the DR public primary school system? 3. What have been the most significant challenges or limitations? 4. Did the project conduct student and teacher performance evaluations in ESP schools? If yes, did you receive information on the outcomes? ! Yes, received information on results of evaluations ! No, did not receive information on results of evaluations Reading 5. During the period of the project (2009-2015), on average, did reading comprehension scores increase in the schools receive interventions from PEF? Boys Girls Both Yes No Yes No Yes No Increased? By how much? 6. What factors have you found that have contributed to low and high reading achievement in ESP schools that have received the interventions? High Reading Achievement Low Reading Achievement 7. Which reading approach (whole language, phonics, balanced, or other) has the program adopted? Do you agree with this approach? Why or why not? Safe Schools 8. What changes have you observed as a result of Safe Schools Activities? (Probe: Is the school safer now for boys and girls? Did the learning environment improve?) Annex D. Data Collection Instruments 57 9. Have you noticed any change in the behavior of boys and girls as a result of Safe Schools activities? 10. Has there been an improvement in attendance and/or academic performance as a result of a safe and improved learning environment? ! Yes ! No Special Needs 11. How did the ESP address the needs of boys and girls with disabilities? 12. Were the activities sufficient to address all the learning needs of children with disabilities? Why or why not? ! Yes ! No If yes, what was the result of the activities? If no, what else is needed? Effectiveness 13. How well was the project implemented, coordinated and managed? What were the strengths? What were the challenges and lessons learned? USAID Coordination 14. Have you received any other training assistance from other USAID projects or other donors? If yes, which ones? Annex D. Data Collection Instruments 58 15. Does ESP coordinate with these projects? If yes, how? 16. What are the advantages of multiple USAID projects operating in the same geographic area? 17. What are the challenges or obstacles to coordination where there are multiple USAID projects (e.g., PUCMM’s ESP and AMCHAM’s BEP) operating in the same area? 18. Does USAID achieve greater impact when projects operate in the same area? (If yes, why? If not, why not?) Final Recommendations 19. If the project were replicated, what recommendations would you propose to USAID or PUCMM for improving the impact of the program? (Probe for reading, safe schools, and inclusive education) 20. Do you have any recommendations for improving the sustainability of the project within the MOE? Note to the evaluator: Complete the checklist for each interview. 1. Has the program achieved its expected results and targets? ! Yes ! No 1c. Were the results of teacher and student performance disseminated to key stakeholder(s) at the national level? ! Yes ! No 3. Did the ESP improve reading skills of primary school boys and girls in targeted communities? ! Yes ! No Annex D. Data Collection Instruments 59 4a. Did the learning environment improve as a result of Safe Schools activities? 4e. If yes, did it have any impact on reading performance? ! Yes ! Yes ! No ! No 5b. Were ESP special needs activities sufficient to address all the learning needs of children with disabilities? ! Yes ! No 5c. Has enrollment of children with disabilities increased? Amount increased (#) ___________________ Amount increased (%) ___________________ ! Yes ! No 5d. Has academic achievement of children with disabilities increased? ! Yes ! No Annex D. Data Collection Instruments 60 SCHOOL PRINCIPAL QUESTIONNAIRE School Name: District: Date: Evaluator: Name of Person Interviewed: Title: Introduction: Thank you very much for your participation in the evaluation. The purpose of the evaluation is to learn about your experience in relation to the Effective Schools program, your opinion about the key accomplishments and lessons learned, and the contributions to the Dominican public primary school system. In particular, we are interested in learning about the results of the Spanish language, safe schools and inclusive education components. You will be asked to make recommendations to improve the ESP program and future USAID/DR programs. Your responses will be confidential. Number of Students Enrolled: Introductory Questions 1. How long have you been in your current position? ________________ 2. How long have you been working with the project? _______________ 3. In which project components and activities have you participated? ! In-service training workshop (Literacy) ! In-service training workshop (Math) ! Management training ! Safe Schools training ! Inclusive education trainings ! All five components ! Other ___________________________________________ Effectiveness 1. What was your opinion of the training? Was it sufficient for you to fully grasp the skills and concepts needed to effectively apply it? (If yes, ask for examples of what they are doing differently as result of the training) 2. In your opinion, what are the most significant accomplishments of the Effective Schools Program (ESP)? What are its main contributions to your schools? 3. What have been the most significant limitations of the program? Annex D. Data Collection Instruments 61 4. Did the project conduct student and teacher performance evaluations in your school? If yes, did you receive information on the outcomes? ! Yes, received information on results of evaluations ! No, did not receive information on results of evaluations Reading 5. During the period of the project (2009-2015), did reading comprehension scores increase for students? If yes, by how much on average? Which grade showed the most improvement between first and fourth grade? Boys Girls Both Yes No N/A Yes No N/A Yes No N/A Increased scores? Percentage increase? Which grade? 6. What factors have you found contribute to reading achievement? (probe for low achievement and high achievement) High Achievement Low Achievement Safe Schools 7. What changes have you observed in the school as a result of the Safe Schools Activity? (Probe: Is the school safer now for boys and girls? Did the learning environment improve?) 8. What changes have you observed in the behavior of boys and girls as a result of Safe Schools Activities? 9. Are children using less abusive language? If yes, could you please explain? ! Yes ! No Annex D. Data Collection Instruments 62 10. Is there a reduction in violent behavior? If yes, could you please explain? ! Yes ! No 11. Has there been an improvement in attendance and/or academic performance as a result of a safe and improved learning environment? ! Yes ! No 12. Has a safe and improved learning environment contributed to an improvement in reading performance? ! Yes ! No ! Don’t know Special Needs 13. How did the ESP project address the needs of boys and girls with disabilities? 14. Were the activities sufficient to address all the learning needs of children with disabilities? Why or why not? ! Yes (If yes, what was the result of the activities?) _____________ ! No, (If no, what else is needed?) ________________________ 15. Has enrollment of children with disabilities increased in your school as a result of ESP activities? (Collect information on enrollment numbers from PUCMM) ! Yes; If yes, how much has it increased? # ____________ % _________ ! No Annex D. Data Collection Instruments 63 16. Has the academic achievement of students with disabilities improved as a result of the ESP activities? ! Yes; How much has performance improved on learning assessments? __ ! No Effectiveness 17. How well was the project implemented, coordinated and managed? What were the strengths? What were the challenges and lessons learned? Final Recommendations 18. If the project were replicated, what recommendations would you give to improve the effectiveness of the program, especially with regards to reading, safe schools and inclusive education? Note to the evaluator: Complete the checklist for each interview. 1. Has the program achieved its expected results and targets? ! Yes ! No 1c. Were the results of teacher and student performance disseminated to school? ! Yes ! No 3. Did the ESP improve reading skills of primary school boys and girls in targeted communities? ! Yes ! No 4a. Did the learning environment improve as a result of Safe Schools activities? 4e. If yes, did it have any impact on reading performance? ! Yes ! Yes ! No ! No 5b. Were ESP special needs activities sufficient to address all the learning needs of children with disabilities? ! Yes ! No 5c. Has enrollment of children with disabilities increased? Amount increased (#) ___________________ Amount increased (%) ___________________ ! Yes ! No 5d. Has academic achievement of children with disabilities increased? ! Yes ! No Annex D. Data Collection Instruments 64 SPANISH LANGUAGE TEACHER QUESTIONNAIRE School Name: District: Date: Evaluator: Name of Person Interviewed: Grade: Introduction: Thank you very much for your participation in the evaluation. The purpose of the evaluation is to learn about your experience in relation to the Effective Schools program, your opinion about the key accomplishments and lessons learned, and the contributions to the Dominican public primary school system. In particular, we are interested in learning about the results of the Spanish language, safe schools and inclusive education components. You will be asked to make recommendations to improve the ESP program and future USAID/DR programs. Your responses will be confidential. Introductory Questions 1. How long have you been in your current position? ________________ 2. How long have you been working with the ESP project? _______________ 3. In which project components and activities have you participated? ! In-service training workshop (Literacy) ! Management training ! Safe Schools training ! Inclusive education trainings ! Other ___________________________________________ Effectiveness 1. What was your opinion of the training? Was it sufficient for you to fully grasp the skills and concepts needed to effectively apply it in the classroom? 1b. If yes, ask for example of what they doing differently as a result of the training. What strategies did you apply that has really helped your students to learn how to read and comprehend well? 2. Did you receive any coaching assistance with implementing the new approaches in the classroom? If yes, from whom and how often? _________ ! Once a week ! Twice a week Annex D. Data Collection Instruments 65 ! Once a month ! Twice a month ! Other ____________ If yes, how would you rate the effectiveness of the coaching assistance on a scale of 1-3 where 3 indicates very useful, 2 useful, and 1 is not very useful? Place an X in the appropriate box. Coaching X Not very useful 1 Useful 2 Very useful 3 3. What type of reading approach do you employ to teach reading (phonics, whole language, mixed whole language and phonics)? Is it effective? Why or why not? Approach X Phonics-based approach (teaches letter sounds and decoding skills – segmenting/blending of sounds and syllables) Whole-language approach (uses rhymes, songs, stories, and context, but does not teach letter sounds, syllables or words in isolation) Both (balanced approach) 4. Do you have any special needs student in your classroom? ! Yes ! No 4b. If yes, what kind of learning or physical disability does the child have? 4c. How do you accommodate the child’s learning needs? 5. How often do you assess your students’ reading skills? ! 5b. Request to see PUCMM/USAID continuous assessment form. 5c. Do you use the results of the formative assessment to plan your lessons? Annex D. Data Collection Instruments 66 Results of Reading 6. Has there been a formal assessment of your students’ reading skills? If yes, how did they perform? ! Yes, performed well ! Yes, did not show any improvement ! No, were not assessed 7. On average, did reading comprehension scores increase during the time period (2009-2015) of the project? (For girls; For boys) Boys Girls Both Yes No N/A Yes No N/A Yes No N/A Achievement Increased? What % improved since beginning of project? Which grade had highest results? 8. What factors have you found that contribute to reading achievement? (Probe for high and low achievement) High Achievement Low Achievement 9. Did the project conduct teacher performance evaluations in your school? If yes, did you receive results of the evaluation? ! Yes, received results of evaluation ! No, did not receive results Accomplishments/Lessons Learned 10. In your opinion, what are the most significant accomplishments of the Effective Schools Project (ESP)? What are its main contributions to your school? Annex D. Data Collection Instruments 67 Recommendations 11. What do you think is needed to continue improving students' reading skills? 12. What do you think teachers need to continue improving their reading instruction methods? 13. If the ESP project were continued or replicated in other schools, what recommendations do you have for improving the effectiveness of the program? Note to the evaluator: Complete the checklist for each interview. 1. Were the results of teacher and student performance disseminated to teachers? ! Yes ! No 3. Did the reading comprehension scores improve as a result of the project? ! Yes ! No 4. Any challenges with coordination of project activities? ! Yes ! No Annex D. Data Collection Instruments 68 SPANISH LANGUAGE CLASSROOM OBSERVATION TOOL 1. Request to observe teacher who has been trained by PEF 2. Select teacher in grades 1-4 depending on school and shift observed. (Vary the grades with emphasis given to 1st and 2nd grade). 3. Prior to classroom observation, request to see teachers’ lesson plan and teachers’ guide and complete Section A of the questionnaire. (NOTE: If not possible, request it after the classroom observation) 4. Observe the class for the full period. During the observation, complete Section B. of the checklist. 5. Towards the end of the class (last 10-15 min.) complete Section C reflecting on Section B and classroom environment. 6. After the observation, complete Section D and identify any gaps. Then prior to the teacher interview, collect any missing information. For example: a. Ask teacher to present lesson plan, teacher guide and assessment form; b. Ask teacher to explain how today’s lesson corresponded with the teacher guide. c. Probe for what reading methodology teacher used today. d. Ask how often they assess students’ reading skills and request to see grade book or assessment form. e. Ask how many students have Spanish textbook/reading material, etc. 7. Most important sections to complete are Sections C and D. Annex D. Data Collection Instruments 69 School Name: ________________ Observer Name: ________________ District: _______________ Date: ________________ Teacher: ________________ Time: ________________ Instructions: Select a 1st- 4th grade class for observation depending on which teacher was trained. If given an option, select 1st or 2nd grade. Ensure that it is a Spanish language class. Observe the class for 30-45 minutes. Complete Section A prior to class, section B during the classroom observation, and section C post-observation. Allow one hour to complete. A. Classroom Demographic Information School Identification Information (complete this before the lesson begins with information from the head teacher and/or teacher) 1. Type of school ! High reading achievement ! Low reading achievement ! 5-Component School ! Safe School ! Extended day ! Inclusive education 2 Grade 3. a. Does the teacher have a lesson plan? a. " Yes b. " No c. " N/A __________________ b. Does the lesson plan include aspects of appropriate reading methodology according to the teacher guide? a. " Yes b. " No c. " N/A __________________ c. Does the teacher have his/her own teachers’ guide? a. " Yes b. " No c. " N/A __________________ 4. Class has learning materials displayed on the walls a. " Yes b. " No Comment on materials displayed (both student and teacher work, sufficient or insufficient): Annex D. Data Collection Instruments 70 B. Classroom Observation General Activities Observed Not observed Listening to story read aloud Teacher asking comprehension questions Student answering questions Students reading aloud Singing rhymes or songs Small group work Clap/stomp the syllables Assessment (written or oral) = students are asked to read and teacher marks in her continuous (monthly/weekly) assessment form the grade of the student; or collects work to mark; Students complete an end of unit test; Relating material to students’ lives and other subjects Providing one-on-one instruction to child (e.g., helps correct child’s spelling or pronunciation while reading) Dictation – Teacher reads aloud and the students write what they hear. (If teacher writes on the board, it is copying not dictation). Drawing/writing/sketching Other: Reading-Specific Activities Phonological Awareness: 1 Making correct letter sounds 2 Identify sounds in spoken words (not written language) (listen to a story and clap when they hear a sound or word) Phonics/Decoding: 3 Differentiating between correct letter name and the correct letter sound (reading the alphabet and then sounding out the alphabet) 4 Reading syllables (one syllable words – el, la, tu, etc.) (two syllable words; “ki-tten”; three or four syllable words) 5 Blending letter sounds “t-oo” = too; or breaking apart words into syllables and sounds= alphabet - “al-pha-bet” – concentrating on “ph” - physics, phone, etc. Vocabulary and fluency: 1 Identifying words in story text 2 Learning new vocabulary words isolated 3 Reading words/sentences from printed material or book 4 Reading words/sentences from the chalkboard Comprehension 1 Listening Comprehension: Students answering questions related to the oral story read aloud by teacher 2. Reading comprehension: Students answer questions based on story they read with class or individually and textbook is in front of them 2 Reading story through pictures 3 Writing/telling their own stories 4 Drawing picture based on story read/heard 5 Predicting story events/outcomes Annex D. Data Collection Instruments 71 What materials did the Teacher Use? 1 Blackboard/Chalkboard all the time 2 Textbook 3 Workbook 4 Supplementary materials (from reading corner) 5 Poster/wall charts (with letters, words, pictures) 6 Audio/visual technology 7 Flash cards 8 Student notebooks 9 Manipulatives (e.g. real objects, bottle caps, clay, sand, cut out word) C. Post-Observation Summary of Observations Instructions: Complete this section from reviewing the results of the observation in Section B and reflecting on the outcomes. 1. What types of activities did you observe the teacher doing in general? ! Relates material to students’ prior knowledge ! Small-group work ! Pair work ! Utilizes supplementary materials from reading corner/library/other ! Utilizes audio/visual technology as a teaching aid ! Classroom management ! Grouping students by ability ! Provides individual instruction or exercises for struggling learners/learners with special needs 2. What types of materials did the teacher use? 3. Which of the five key reading skills did you observe the teacher teaching? Reading Skills √ 1 Phonological awareness: Practicing correct letter sounds 2 Phonics: Teaching the alphabet names and sounds, reading letters Phonics: Reading syllables (seeing words made up as syllables e.g., kit-ten) (“beat the word” – clap, beat/tap/stamp the syllables of the word) Decoding: Blending letter sounds to make words, breaking apart whole words 3 Vocabulary: Reading key familiar/sight words 4 Fluency: Reading connected text in authentic context that has meaning (reading a story, newspaper, book) (Opposite: reading unrelated vocabulary words) 5 Reading Comprehension: Uses pictures and prior knowledge to aid comprehension and predict story events and outcomes; guided practice of reading and comprehension; teacher asks students to read and then asks questions about what they read. (Text is in front of child) 6 Oral Listening Comprehension: Pre-reading skill. Listening to story and answer Annex D. Data Collection Instruments 72 questions. (No text in front of child) 4. What types of activities did the teacher use to promote reading in the classroom? Check all that apply: ! Rhymes, songs, role-playing ! Teacher reading story aloud ! Students reading with text in front of them ! Students reading sentences ! Students reading whole stories/books independently ! Reading in pairs ! Students writing own stories ! Doing exercises in groups ! Asking reading comprehension questions ! Asking listening comprehension questions ! Post-lesson evaluation (written reflection/assessment/drawing) 5. How many learners have a Spanish language textbook? ! All students ! The majority of students ! Half of the students ! Less than half ! None ! Did not see language books 6. How many learners have a notepad and pen/pencil? ! All students ! The majority of students ! Half of the students ! Less than half ! None ! N/A 7. Does the classroom have a library/reading corner? ! Yes ! No 8. Are there any observed constraints affecting teaching and learning of reading? (large class size, lack of reading materials, student physical/learning disability, arrangement of the room, etc.) 9. Are there any other observations or comments? Annex D. Data Collection Instruments 73 Section D. Post-Observation Checklist Tick the appropriate box below to assess the 6Ts of Effective Language Instruction and provide any additional comments or observations. Note to Evaluator: Complete the Classroom Observation Summary Below 6Ts of Effective Language Instruction Observed Comments Teacher – Did the teacher following the lesson plan in the teacher’s guide? ! Yes No Tongue – Did the teacher teach in the local language spoken by the children? ! Yes No Text – Did pupils read from printed material? ! Yes No Time – Did the teacher use the entire class time for literacy reading and writing? ! Yes No Test – Is the teacher assessing students using a Continuous Assessment or other Form? (Note: This question is asked in the teacher interview – please record the response here as well) ! Yes No Teaching methodology - Is the teacher using appropriate modeling and reading methods? (models correct pronunciation; explicitly shows students how to decode a word into syllables and sounds, points at the words from left to right, follows the suggested reading approaches in the teacher’s guide) ! Yes No Factors observed affecting reading achievement of students? (large class size; lack of reading materials; student physical/learning disability; arrangement of the room, lack of classroom displays to stimulate learning; language barriers for non-Spanish speakers, insufficient notepads/pens/workbooks; etc.) ! Yes No Annex D. Data Collection Instruments 74 SAFE SCHOOLS SCHOOL STAFF QUESTIONNAIRE School Name: District: Date: Evaluator: Name of Person Interviewed: Title: Introduction: Thank you for your participation in this evaluation. The purpose of the evaluation is to learn about your experience in the Safe School activity/component, your opinion about key accomplishments, lessons learned, and how it has helped create a positive learning environment for students. We would like to learn about the Safe Schools effect on learning and achievement. You will be asked to make recommendations to improve the Safe Schools Program and for future USAID/DR programs. Your responses will be confidential. Number of People Interviewed: ______ Introductory Questions 1. What is your current role in the school? _______________________ 2. How long have you been in your current position? ________________ 3. How long have you been working with the Safe Schools activity? __________ 4. Which activities prepared you to work in the Safe Schools activity? 5. What was your opinion of the training? Was it sufficient for you to fully grasp the skills and concepts needed to effectively carry out your role in the Safe Schools program? ! Yes ! No 6. Did you develop an action plan at the training to implement Inclusive Education in your school? If yes, what specific activities did you plan for your school? Primary Questions Annex D. Data Collection Instruments 75 1. What kinds of activities did you conduct to improve the learning environment of students after the training? 2. What kinds of issues related to violence or abuse have been identified? 3. How are the cases reported and addressed? 4. What are the most significant changes you have observed in students’ as a result of the Safe Schools interventions? 5. Has there been an improvement in attendance as a result of the Safe Schools activities? ! Yes ! No ! Not a problem ! Don’t know 6. Has academic achievement of students improved as a result of the Safe Schools activities? (Probe for evidence) ! Yes ! No ! Don’t know 7. a. If yes, has there been any improvement in reading and writing, specifically? b. If not, what are the factors contributing to low student performance? 8. Which of the Safe Schools activities has had the greatest impact on improving the safety and learning environment for students? Annex D. Data Collection Instruments 76 9. Has the Safe Schools activities had any impact on enrollment of children in the school? (e.g., on children with disabilities, young children, girls, etc.) Parents and Community Involvement 10.What types of community organizations or parents’ groups do you coordinate with? a. Mothers and Fathers Association (If yes, how?) b. Legal representatives c. Community organizations d. Other____________ 11.Have there been any discussions with the parents and community members on the issue of violence in your school? If yes, how did you engage parents in the topic? 12.Is the concept of GBV understood and accepted by parents, relatives/guardians, and community members who support Safe Schools? ! Yes (Explain ____________________ ! No (Explain ______________________ 13.How are Safe Schools activities promoted within the community? Probe - What is the impact? ! School newsletters ! School events, student performances, school festivals, book fairs, etc., ! Parent –teacher conferences ! Community events ! Meetings with local judiciary representative ! Other (Specify _______________________________________ Final Recommendations 14.If the project is replicated, what recommendations do you have for improving the Safe Schools activity? Probe for specifics. Note to the evaluator: Complete the checklist for each interview. 4a. Is the school safer? Did learning environment improve? ! Yes ! No Annex D. Data Collection Instruments 77 4b. Are children using less abusive language? ! Yes ! No 4c. Is there a reduction in violent behavior? ! Yes ! No 4d. Has there been an improvement in attendance and performance as a result of a safe and improved learning environment? ! Yes ! No 4e. Has a safe and improved learning environment contributed to an improvement in reading performance? ! Yes ! No Annex D. Data Collection Instruments 78 INCLUSIVE EDUCATION SCHOOL STAFF QUESTIONNAIRE School Name: District: Date: Evaluator: Name of Person Interviewed: Title: Introduction Thank you for your participation in this evaluation. The purpose of the evaluation is to learn about your experience in the Effective Schools Program (ESP) and your opinion about key accomplishments and lessons learned for creating a positive learning environment for children with special needs. We are interested in understanding the impact of the ESP Inclusive Education activities on student enrollment and achievement. I will ask you to make recommendations to improve Inclusive Education and for future USAID programs in the Dominican Republic. Your responses will be confidential. We will begin with general questions. Introductory Questions 1. How long have you been in your current position? ________________ 2. How long have you been working with the ESP? _______________ 3. In which program components and activities have you participated? ! Literacy training ! Math training ! Management training ! Safe Schools training ! Inclusive education training | # of trainings: ______________ ! Other: _______________ Activities 1. What was your opinion of the Inclusive Education training? What did you learn? Was it sufficient for you to fully develop the skills needed to carry out your role in Inclusive Education effectively? Annex D. Data Collection Instruments 79 2. Did you develop an action plan at the training to implement Inclusive Education in your school? If yes, what specific activities did you plan for your school? 3. What support has the ESP provided to help students with special needs at this school? Probe for specifics. ! Diagnostic testing ! Referral to specialist ! School staff training ! School staff mentorship ! Provision of materials ! Awareness session ! Database with local resources ! Other: ______________ 4. Who was involved in the diagnostic process? Did you receive the results? If so, from whom and when? Diagnostic Test From Whom (PUCMM or Ministry) Month Year First Second 5. Did the school collaborate with any other organizations to provide students at this school with professional diagnosis and treatment? If so, were there challenges to coordination with partners? ! Yes | Organizations: _____________________ ! No 6. What types of disabilities and learning difficulties do students at this school have? Type # of students detected # of students diagnosed # of students treated Comments Physical Hearing Annex D. Data Collection Instruments 80 Visual Intellectual Other: 7. Does this school have an Espacio de Apoyo? If so, how do teachers and students use the Espacio de Apoyo? On average, what percentage of the day do teachers and students utilize the room? Results 8. How have you changed the way you support children with special needs as a result of the ESP? Could you provide an example? 9. Have other teachers, school staff, or parents changed the way they support children with special needs as a result of the ESP? Could you provide an example? 10. Have you observed any changes in students with special needs as a result of ESP’s Inclusive Education activities? ! Participation in the classroom ! Self-esteem ! Socialization on school grounds, e.g., during recess ! Discrimination, e.g., bullying ! No improvement ! Does not know ! Other: ______________ 11. Has academic achievement of children with disabilities improved as a result of Inclusive Education activities? If yes, what factors have contributed to this improvement? ! Yes, general performance improved ! Yes, reading performance improved ! No, general performance did not improve Annex D. Data Collection Instruments 81 ! No, reading performance did not improve ! Does not know 12. Has enrollment of children with disabilities increased as a result of Inclusive Education activities? If yes, what factors contributed to this increase? ! Yes, enrollment has increased | # of new students: ______________ ! No improvement in enrollment ! No problem with enrollment prior to the esp ! Does not know 13. In your opinion, what are the most important contributions of the Inclusive Education activities to improving conditions for children with special needs? Challenges 14. What have been the most significant challenges to the implementation of the Inclusive Education activities in this school? 15. Was the ESP able to address all of the learning needs of children with disabilities? If not, what is the most critical type of support that the school needs to support the learning of students with special needs? ! Yes ! No 16. What are the main barriers to school enrollment for children with disabilities and learning difficulties? 17. What are the main barriers to learning for children with disabilities and learning difficulties? Annex D. Data Collection Instruments 82 Recommendations 18. What recommendations would you propose for increasing the impact of the Inclusive Education activities? Probe for specifics. Note to the evaluator: Complete the checklist for each interview. 1. Did the project adequately address the learning needs of students with special needs? ! Yes ! No 5c. Has enrollment of children with disabilities increased? Amount increased (#) ___________________ Amount increased (%) ___________________ ! Yes ! No 5d. Has academic achievement of children with disabilities increased? Evidence: ________________________ ! Yes ! No 6. Any challenges with implementation of Inclusive Education activities? ! Yes ! No Annex D. Data Collection Instruments 83 PARENTS FOCUS GROUP TOOL School Name District Date Evaluator # of parents interviewed # of males: # of females: Instructions for Evaluator: Read the introduction and introductory questions. Depending on which activities parents have participated, ask questions about their involvement for that specific component. For all parents, ask about parental involvement, situation of special needs and violence to know if it is a problem at that school, and seek recommendations for improving the reading, school environment or for addressing special needs (if applicable). Introduction: Thank you for your participation in the evaluation of the ESP. I would like to discuss your experience in relation to three activities of Effective Schools Program/EFP. The specific components are Reading, Safe Schools, and Inclusive Education. Furthermore, we would also like to learn about other activities that your organization/group is involved in at this school as well as recommendations for how to improve the quality of education. Introductory Question 1. Which groups are present? Group Role Mothers’ and Fathers’ in Action Comite de Curso School Board Other: 2. What is your understanding of the Effective Schools Program/ESP? What do you think is the objective? 3. Which activities have you participate in within the ESP project? ! Reading ! Safe Schools ! Inclusive Education ! All three components ! Other ___________________________________________ 4. Read the questions for relevant components and record comments. Ask about the situation of Inclusive Education and Violence in the community if the parents are only involved in the reading activities. All parents should be asked about how to improve parental involvement and recommendations for improving the quality of education. Annex D. Data Collection Instruments 84 General 1. What kinds of activities has your group conducted to support the school? Reading 2. What kinds of meetings, trainings, or events related to reading have you participated? (e.g., reading fairs, how to support children with homework, assisting in the classroom) 3. Have you had any training or discussion with teachers or the principal about how to support your child’s reading? (If yes, what did you learn?) 4. How have you applied what you have learned about supporting your children’s reading at home? 5. Have you observed a difference in your child’s interest or motivation to read? If yes, request example 6. Have you noticed any improvement in their ability to read and write? If yes, what? Safe Schools 7. What kinds of meetings, trainings or events related to Safe Schools have you participated? 8. Have you noticed any difference in the safety of children at school or a reduction of violence? If yes, probe for before and after. 9. What can you tell me about the current situation of violence in the school or community and about how it impacts children at school? Inclusive Education 10. Are there any children with disabilities at the school? If yes, how many? 11. Has there been any improvement in treatment or perceptions of children with disabilities? If yes, what has contributed to this improvement? 12. Do you know if there are children with disabilities who are out of school? How many? What is your opinion of the education of children with special needs? (e.g., should they be in the regular school?) Has there been any increase the number of children with disabilities enrolled in this school? If yes what has contributed to this improvement? Parental Involvement 13. What percentage of parents regularly participates in school activities? (meetings, discussions, etc.) 14. How often do parents speak to the teachers about their children’s academic performance? Has there been any improvement in the level of communication between parents and teachers? If not, why not? Recommendations 15. What recommendations do you have for involving more parents in activities that support children’s reading skills, increase their safety or address their special needs? 16. Do you have any other comments or recommendations to improve the quality of education? Annex D. Data Collection Instruments 85 YOUTH FOCUS GROUP TOOL Tool: Highly interactive focus group activity using a simple Past-Present-Future framework. Purpose: To engage youth in an activity to provide input regarding the Safe Schools activity. Facilitation: Interviewer is principal facilitator and will request volunteer facilitators and documenter/recorder for sub-groups. Material: Post-it notes, flipchart paper, flipchart stand writing pens, scotch tape. Introductory Protocol: My name is ______ and I am with the USAID Safe Schools Program. Thank you for being here. I am very happy to see you her because your ideas will help to make safe schools activities better in the future. What you discuss during this discussion will be confidential. No names will be attached to any comments. I am here to listen to your opinions about the Safe Schools program and how it can be improved for the safety of all children. Your information will be used to for future plans that will serve school children in their efforts to achieve their education goals. I would like to have your permission to record this activity either on the flipchart paper, post-it notes or on my iPhone. Do I have your permission? Thank you. Let’s begin! Steps: 1. Explain this is a participatory process and everyone’s input is important. 2. Explain the “Past-Present-Future” framework. 3. They are asked to write one comment per post-it note about each category they can relate to/have experience with. Instruct that they do not write on the “sticky” side as they will attach to a larger sheet of paper. 4. Engage students to help you. 5. Discuss the topic then ask them to write one comment per post-it note. Then “stick” their post-it notes onto the appropriate category. 6. Provide time for general comments. Take a photo. 7. Interviewer/facilitator will provide a quick summary of the group’s input. 8. Conclude by asking students for ideas/action plan for solving the issues they presented. 9. Thank the group for their participation and explain that their comments will be used improve their school for current and future programming. Framework: Topic Past Present Future Gender-based Violence Verbal Abuse Education for Special Needs Students/Inclusive Education Annex D. Data Collection Instruments 86 How to Guide the Discussion: Topic Past Present Future Appropriate words to use SAFE SCHOOLS What does “Escuela Segura” mean to you? What activities were you involved in? What was your school like in the past? What is your school like now? How would you like your school to be like in the future? GENDER-BASED VIOLENCE What does gender￾based violence mean to you? Please give me examples of gender￾based violence. What type of gender-based violence has happened in the past at school or in your community? What type of gender-based violence still exists? How could you imagine your school in the future without gender based violence? If they do not understand gender violence: men abusing women; women abusing men physically, verbal, emotionally. VERBAL ABUSE What does verbal abuse mean to you? Please give me an example of verbal abuse. What type of verbal abuse happened in your school in the past? What type of verbal abuse is still happening in your school? What would a school without verbal abuse be like? Annex E. Sources of Information 87 ANNEX E. SOURCES OF INFORMATION REFERENCE LIST ESP Cooperative Agreement Reports PUCMM, Effective Schools Program Cooperating Agreement Report: October 1, 2010 – December 31, 2011, prepared for USAID. PUCMM, Annual Progress Report: October 2011 – September 2012, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Quarterly Report (Q1): October 1, 2009 – December 31, 2009, prepared for USAID PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Quarterly Report (Q2): January 1, 2010 – March 31, 2010, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Quarterly (Q4) Report – Annual: July 1, 2010 – September 30, 2010, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Quarterly Report (Q2): October 1, 2010 – December 31, 2010, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: January 1, 2011 – May 31, 2011, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: April 1, 2012 – June 30, 2012, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: July 1, 2012 – Sept 30, 2012, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: October 1, 2012 – December 31, 2012, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: January 01, 2013 – March 30, 2013, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: January 1, 2014 – March 31, 2014, prepared for USAID. PUCMM, Effective Schools Program (USAID – PUCMM) Cooperating Agreement Report: April 1, 2014 – June 30, 2014, prepared for USAID. Other ESP Project Documents PUCMM, 2013, Effective Schools Program Performance Monitoring Plan, prepared for USAID. PUCMM, 2015, Presentación del Programa de Escuelas Efectivas y de la Política de Apoyo a los Primeros Grados: Componentes de Gestión, Lengua Española, Matemática, Escuelas Seguras, Educación Inclusiva, Monitoreo y Evaluación, PowerPoint presentation, prepared for USAID final closing ceremony. PUCMM, 2015, Impacto del Programa de Escuelas Efectivas en los Aprendizajes de Matemática y Comprensión Lectora 2013 y 2014, PowerPoint presentation, prepared for USAID final closing ceremony. USAID/Dominican Republic, 2013 Education Portfolio Mid-Term Performance Evaluation, http://pdf.usaid.gov/pdf_docs/pdacu985.pdf. USAID/Dominican Republic, Effective Schools Program Cooperative Agreement with PUCMM. USAID/Dominican Republic, Effective Schools Program Cooperative Agreement Modification 5 with PUCMM. USAID/Dominican Republic, Effective Schools Program Cooperative Agreement Modification 6 with PUCMM. USAID/Dominican Republic, Effective Schools Program Cooperative Agreement Modification 8 with PUCMM. Annex E. Sources of Information 88 Reading Documents PUCMM, 2014, Analisis del Impacto del Programa de Escuelas Efectivas – PEF Lengua, prepared by Gilbert Valverde, Eduardo Luna, Renzo Roncagliolo, and Joel Zapata. PUCMM, 2014, Programa de Escuelas Efectivas: Analisis del Impacto del Programa de Escuelas Efectivas – PEF – en el Area de Comprension Lectora 2010-2012, prepared by Gilbert Valverde, Eduardo Luna, Renzo Roncagliolo, and Joel Zapata for USAID. PUCMM, 2014, Programa de Escuelas Efectivas: Analisis del Impacto del Programa de Escuelas Efectivas – PEF – en el Area de Comprension Lectora 2013-2014, prepared by Gilbert Valverde, Eduardo Luna, Renzo Roncagliolo, and Joel Zapata for USAID. PUCMM, Centro de Excelencia para Capacitación de Maestros de Centroamérica y República Dominicana (CETT/CA-RD): Plan Operativo de Capacitación de Capacitadores (POC￾Capacitadores-Regional). PUCMM, Centro de Excelencia para Capacitación de Maestros de Centroamérica y República Dominicana (CETT/CA-RD): Plan Operativo de Capacitación Regional (POC-Regional). PUCMM, Plan Operativo De Acompañamiento (POAC) CETT/CARD. PUCMM, El Modelo de Desarrollo Profesional de Docentes del Programa Escuelas Efectivas – Lengua Española, PowerPoint presentation. PUCMM, Indicadores Relacionados con la Gestión Pedagógica, Afectiva y Curricular. PUCMM, Package of Student Math and Language Tests. USAID, 2011, Centers for Excellence in Teacher Training (CETT) Program: Paper Three: Sustainability, Final Report, White Paper Series, prepared by JBS International for USAID. USAID, 2012, Centers for Excellence in Teacher Training (CETT) Program: Paper Two: Teasing and Assessment, White Paper Series, prepared by JBS International for USAID. USAID, 2013, EdData II: Task Order 7: Learning Outcomes Research and Assessment-Related Projects, Final Report, prepared by RTI International for USAID. USAID, 2013, EdData II: Resultados de Rendimiento en la Lectura y Matemáticas Así Como Hallazgos de las Prácticas Pedagógicas y de Gestión Escolar en una Muestra de Escuelas de la República Dominicana, prepared by RTI International for USAID/Dominican Republic and AMCHAM. USAID, 2013, EdData II: Results of EGRA/EGMA Application in Participating Schools of the Basic Education Program – USAID and AMCHAM Dominican Republic, prepared by RTI International for USAID/Dominican Republic and AMCHAM. USAID, 2015, Are Latin American Children’s Reading Skills Improving? Highlights from the Second ad Third Regional Comparative and Explanatory Studies (SERCE and TERCE), prepared by Dr. Alejandro Ganimian for American Institutes for Research for USAID, draft. Safe Schools Documents DevTech Systems, USAID Safe Schools Program (2003-2008), http://www.devtechsys.com/practices/education-youth-development/43-safe-schools-program PUCMM, 2014, Programa de Escuelas Efectivas PEF (USAID – PUCMM) Componente Escuelas Seguras: Exploración de Impacto Programa Escuelas Seguras, October. PUCMM, 2014, Safe Schools Program Results, November 25. PUCMM, Package of Workshop Materials. PUCMM, Programa de Escuelas Efectivas PEF Componente Escuela Segura USAID – PUCMM: Encuesta sobre la Violencia en la Escuela (VE): Cuestionario de maestros y maestras. USAID, 2008, Safe Schools Program Final Report, prepared by DevTech Systems for USAID, November, http://www.devtechsys.com/images/eyd/safe-schools-final-report.pdf. USAID, 2013, EdData II: Literature Review on the Intersection of Safe Learning Environments and Educational Achievement, prepared by RTI International for USAID, October. USAID, 2014, Ministry of Education, and PUCMM, Escuelas Seguras, final PowerPoint presentation, October 28. Annex E. Sources of Information 89 USAID, Abriendo Puertas 1, http://www.usaid.gov/sites/default/files/documents/1865/Spanish_Manual_I_508.pdf. USAID, Abriendo Puertas II, http://www.usaid.gov/sites/default/files/documents/1865/Spanish_Manual_II_508.pdf. USAID, Abriendo Puertas III, USAID, Abriendo Puertas 1, http://www.usaid.gov/sites/default/files/documents/1865/Spanish_Manual_III_508.pdf USAID and PUCMM, 2014, Programa de Escuelas Efectivas PEF (USAID – PUCMM) Componente Escuelas Seguras: Percepción sobre Violencia Escolar en Escuelas Públicas del Distrito Educativo 06–03 Jarabacoa: Informe de Resultados, April. Inclusive Education Documents Ministry of Education Directorate of Special Education, 2013, Informe Preliminar sobre Levantamiento de la Información de Estudiantes con Discapacidad en las Escuelas Eficaces, October. PUCMM, Especialidad en Inclusión Educativa de Niños y Adolescentes con Discapacidad. USAID/Dominican Republic, 2014, Inclusive Schools Program Results, draft not for release, December 16. USAID, 1997, USAID Disability Policy Paper. USAID, Ministry of Education, and PUCMM, 2015, Programa de Escuelas Efectivas: Puntos Fuertes del Modelo de Educación Inclusiva, March 27. USAID, Ministry of Education, and PUCMM, Programa de Escuelas Efectiva, Educación Inclusiva, Evaluación de Contexto Escolar. USAID, Ministry of Education, and PUCMM, Programa de Escuelas Efectiva, Educación Inclusiva, Ficha – Consulta Medica. USAID, Ministry of Education, and PUCMM, Programa de Escuelas Efectiva, Educación Inclusiva, Ficha – Necesidad de Apoyo. USAID, Ministry of Education, and PUCMM, Programa de Escuelas Efectiva, Educación Inclusiva, Ficha – Referimiento Medico. USAID, Ministry of Education, and PUCMM, Programa de Escuelas Efectiva, Educación Inclusiva, Ficha – Verificación de Datos. Gender Documents USAID, 2012, Gender Equality and Female Empowerment Policy, http://pdf.usaid.gov/pdf_docs/PDACT200.pdf USAID/Dominican Republic, 2012, DR Gender Analysis: Desk Review Report, Sensitive but Unclassified. USAID/Dominican Republic, 2013, TDY Report and Gender Action Plans for DO Teams, prepared by Regina Jun for USAID. Miscellaneous Documents MINERD-BID, 2013, Linea Base para la Sistematizacion, Componente de Apoyo a los Aprendizajes en Lectoescritura y Matematica, Programa de apoyo al Plan Decenal de Educacion. USAID, 2011, Evaluation Policy, http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf USAID/Dominican Republic, 2013, Country Development Cooperation Strategy FY 2014-2018, http://www.usaid.gov/sites/default/files/documents/1862/Dominican%20Republic%20CDCS_publi c%20version_FY14_FY18.pdf. USAID/Dominican Republic, List of a Few Prior USAID Projects in Education. USAID/Dominican Republic, Request for Applications for Lighting Excitement for Excellency in Reading (LEER), http://www.grants.gov/web/grants/search-grants.html?keywords=dominican%20republic. Annex E. Sources of Information 90 KEY INFORMANTS The table below summarizes the types of respondents from the final performance evaluation of the ESP. Table 9. Respondent Summary Type Total National 15 District 4 Mentor 1 School Principal 52 Spanish Language 49 Safe Schools 26 Inclusive Education 26 Parents 165 Youth 147 Total Respondents 485 SITES VISITED The tables below summarize the schools visited by region and school characteristics. Table 10. Schools Visited by Region Region Schools Jarabacoa 12 La Vega 11 Moca 1 Puerto Plata 1 Santiago 19 Santo Domingo 9 Total Schools 53 Table 11. Types of Schools Visited School Type Schools 5-Component 11 High Reading Achievement 14 Low Reading Achievement 15 Total Reading 49 Safe Schools 25 Inclusive Education 22 Extended Day 6 Annex F. Disclosure of Conflict of Interest Forms 91 ANNEX F. DISCLOSURE OF CONFLICT OF INTEREST FORMS TEAM LEADER – BRENDA SINCLAIR Annex F. Disclosure of Conflict of Interest Forms 92 BASIC EDUCATION SPECIALIST – JUANITA CAMPOS Annex F. Disclosure of Conflict of Interest Forms 93 PROGRAM ANALYST – MARISSA KIMSEY Annex F. Disclosure of Conflict of Interest Forms 94 LOCAL SPECIALIST – AIDA HERNÁNDEZ 95 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523