January 2013 This publication was produced for review by the United States Agency for International Development. It was prepared by Amex International and its subcontract, DevTech Systems, Inc. under Contract No. RAN-I-00-09-00008, Task Order No. AID-517-TO-12-00001. EVALUATION USAID/DOMINICAN REPUBLIC EDUCATION PORTFOLIO MID-TERM PERFORMANCE EVALUATION ANNEXES USAID/DR Effective Schools Program Mid-Term Evaluation Listing of Annexes Annex A: USAID/DR Effective Schools Program (ESP) Mid-Term Evaluation Annex B: USAID/DR The Effective Schools Program (ESP) – Analysis of the Program Database Annex C: USAID/DR Basic Education Program (BEP) Mid-Term Performance Evaluation Annex D: Out of School and At Risk Youth and Children Programs (ARCY) Annex E: Individuals Consulted Annex F: Methodology Annex G: Scope of Work AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex A: ESP i Annex A: USAID/DR Effective Schools Program (ESP) Mid￾Term Evaluation Submitted to: USAID/Dominican Republic Prepared by: Megan Gavin, Evaluation Specialist, ESP Edited by: Virginia Lambert, Team Leader for Portfolio Evaluation January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex A: ESP ii USAID/DR Effective Schools Program Mid-Term Evaluation Annex A: Table of Contents 1. Introduction ............................................................................................................................................. 1 1.1 Background and Description of the Program Evaluated .................................................................... 1 1.2 Methodology ....................................................................................................................................... 2 2. Performance ............................................................................................................................................ 2 2.1 In-service teacher training and curriculum development.................................................................... 2 2.1.1 In-service literacy teacher training and curriculum development.................................................... 2 2.1.2 In-service mathematics teacher training and curriculum development........................................... 6 2.2 Education Management Component................................................................................................10 2.3 Safe Schools program......................................................................................................................11 2.4 Monitoring and Evaluation................................................................................................................12 3. Efficiency ...............................................................................................................................................13 4. Sustainability.........................................................................................................................................15 5. Recommendations ................................................................................................................................18 6. Final Thoughts ......................................................................................................................................20 Appendix A -1. Map of Expansion, source: MINERD, 2011...................................................................21 Appendix A-2. Description of Interventions: ESP, POVEDA, OEI........................................................22 Appendix A-3. List of Key Informant Interviews ....................................................................................24 Appendix A-4. List of Schools visited.....................................................................................................26 Appendix A-5. Classroom observations (data related to Literacy subcomponent)...........................27 Appendix A-6. Classroom Observations (data related to Mathematics subcomponent)..................29 Appendix A-7. Bibliography.....................................................................................................................31 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 1 1. Introduction The DR/ USAID Effective Schools Program (ESP) Mid-Term Evaluation is designed to provide insight on the performance, sustainability and efficiency of the ESP program. The report presents findings and conclusions within each of these domains. This document also serves as a point of reflection to inform the next half of the ESP and provides recommendations for USAID and implementing partner(s) to consider. At the same time, this report contributes to the Education Portfolio Evaluation Integrated Report (Integrated Report). Within the Integrated Report the ESP’s primary focus is in the cross cutting component of the formal education sector which reflects (1) teacher training and coaching, (2) teaching materials, (3) school management. Ultimately, with the USAID Strategy (2011-2015) on the horizon, final thoughts are provided in terms of reaching the Goal of, “improved reading for children in primary grades by 2015.” 1.1 Background and Description of the Program Evaluated The education system in the Dominican Republic consists of the Ministry of Education (MINERD), the Ministry of Higher Education, Science and Technology and the National Institute for Technical Formation. Through a national forum, these three institutions created the National Education Plan (2008-2018). Within the education system, the MINERD is responsible for executing the General Law of Education, 66- 97, which articulates; “education is the permanent right of being human” (article 4). Within this context the ESP program operates in the basic education cycle, specifically in the first cycle (grades 1-4). The MINERD’s current strategy calls on the ESP to expand to the Northern region of the country; see Appendix A-1 (MINERD, 2011). The policy identifies three interventions: the ESP, the Cultural Center POVEDA, and the Organization of Ibero-American States (OEI) to support basic education services via strategies in teacher training in mathematics and literacy, the provision of resources and education management support throughout the DR; see Appendix A-2 for program descriptions. The ESP is implemented via a Cooperative Agreement ($US16.2 Million, 2009-2014) with the Pontifica Universidad Católica Madre Maestra (PUCMM). The ESP program builds on the Centers for Excellence in Teacher Training (CETT) and the Activities to Improve the Effectiveness of Teachers (TEF), which were USAID education programs implemented in the DR in the early 2000s. Whereas the CETT (2002) focused on literacy in grades 1-3, the TEF (2006) focused on mathematics in grades 1-4 and extending literacy work to grade 4. The ESP responds to the demand for quality education. The overall program objective is: “to significantly contribute to the improvement of quality learning of students in the first cycle of primary education.” (p.11, USAID, 2010). To meet this objective the program works in three component areas: • In-service Literacy and Mathematics Teacher Training and Curriculum Development; • Education Management; and, • Monitoring and Evaluation. The full intervention, in other words, all three program components, are implemented in La Vega, Jarabacoa, Santo Domingo Oeste, Herrera and Santiago, Santiago; other selected districts receive one or more intervention(s). Districts are selected in coordination with the MINERD. The Safe Schools Program, which works in grades beyond the first cycle of primary, was piloted in two districts. See the first column in Appendix A-2 for an overview of ESP materials and strategies. In section 2 below, a brief description is provided before the analysis of performance for each component. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 2 1.2 Methodology For a detailed description of the methodology employed in the USAID/DR Midterm Portfolio Performance Review see the Integrated Report. Data specific to this individual report includes: 30 key informant interviews, 42 classroom observations (30 minutes each on average), 24 director interviews, 28 teacher interviews, 3 group interviews with youth from the Safe Schools program and 3 group interviews with parents from the ESP schools (see Appendix A-3 for list of key informant interviews, and Appendix A-4 for a list of schools). Key informant interviews and group interviews were recorded and coded into general themes. Similarly, teacher and director interview responses were grouped by question and responses consolidated. Classroom observation data was tallied and percentages calculated; they were differentiated between duration in mathematics and literacy interventions when applicable. Also unique to the ESP methodology is the review of student achievement data from student assessments administered between 2008 and 2011. This data includes a control sample in 2010 and 2011. Analysis can be found in Annex B. 2. Performance The portfolio review employs the following definition of performance; “the contribution of each component to quality basic education, benefiting girls and boys equally” (AMEX International, 2012). Guiding questions include: Which components are being implemented and where?, How are they being implemented?, What are the characteristics?, Who is benefiting and to what degree?, and Is the project on track to meeting targets? In order to answer these questions and produce clear findings, conclusions and recommendations this section examines each of the ESP’s components. Based on project documentation (quarterly reports, annual reports and implementation plans) outputs are identified. The analysis focuses on the main intervention: “In-service training and curriculum development,” and then on the components: “Education Management”, “Safe Schools”, and “Monitoring and Evaluation.” Findings from performance serve a foundation on which to build the discussion about efficiency and sustainability. 2.1 In-service teacher training and curriculum development The “In-service teacher training and curriculum development” component of the ESP encompasses the literacy and the mathematics subcomponents; their purpose is to “contribute to the improvement of quality education through requiring teachers and technical staff to be effective in achieving an impact in mathematics, reading and writing competencies in first through fourth grade.” (p. 6, ESP, 2012b) 2.1.1 In-service literacy teacher training and curriculum development Output 1: The reading and writing literacy competencies (concepts, procedures, attitudes) of trained teachers and technical staff are strengthened. Finding #1: Teachers participate in trainings and innovation circles; they primarily attribute their growth in competencies to accompaniment. Finding #2: The project’s technical staff engage in a continuous process of professional development to strengthen competencies. Conclusion #1: Classroom level support visits are important to developing teacher competencies. Conclusion #2: The district level technicians have not developed competencies on par with project staff in order to assume the support role. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 3 While teachers participate in trainings and Innovation Circles1 , and attribute these activities to their development of literacy competencies, they primarily attribute their growth to accompanying visits2 from technical staff. A challenge identified in key informant interviews and reiterated by teachers is that they often feel isolated when implementing new methodologies in their classrooms. The accompaniment support they receive mitigates feelings of isolation and facilitates their ability to implement the strategies they learn. As one teacher noted, “the project technical staff is educated and professionally prepared, every time they come they give us follow-up; they are not here to judge us, but to help us.” In addition, the project’s technical staff engage in a continuous process of professional development to strengthen competencies. This professional development appears to be different from the district technician’s competency development. The high quality expertise and commitment of the project technical staff is recognized by teachers, directors, and even district technicians. Teachers and directors, differentiate between the support they receive from district technical staff and project technical staff. One director commented; “The project technical staff provide visits that actually give results, compared to the district technical staff that come once every three months.” Classroom level support by quality technicians is essential to developing teachers’ competencies. The district level technical staff have not developed competencies on par with project level technical staff. What’s more, in order to support teachers in the development of their competencies, it appears unlikely that district level technical staff will be able to play this crucial supporting role. It is important to note that the findings and conclusions on teachers’ competencies are based exclusively on interview data from teachers, directors, and key informants. There are no assessments to date which measure changes in teachers’ literacy competencies. Findings discussed in relation to Output 2 below focus on the application of teachers’ competencies. Output 2: The pedagogical and methodological strategies of teachers trained are consistent with the curriculum proposed by the MINERD and applied in the classroom for strengthening student learning. Finding #3: The MINERD curriculum was developed with guidance from the CETT and ESP programs. Finding #4: Teachers and administrators perceive that students are motivated and interested to learn reading and writing, they suggest this leads to better results. Assessments indicate increases in performance with some exceptions. Finding #5: Teachers and administrators perceive that students are learning how to think for themselves, analyze, infer, and communicate effectively. Conclusion #3: Teachers and administrators perceive the literacy component intervention strategies as effective in strengthening student learning and assessment results largely confirm that they do. 1 Innovation Circles are capacity building activities facilitated by project technical staff at the school level. They are held on a monthly basis for a portion of the school session (maximum 4 hours). They address specific teaching-learning needs and encourage teachers to share practices. The ESP includes 24 hours of Innovation Circle participation. 2 Accompaniment usually entails observation by technical staff of teachers teaching in the classroom. Technicians use an observation tool to determine teacher progress in the ESP strategies. They share their observations with teachers and provide feedback. Teachers receive, on average, 1-2 visits per month per subject area. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 4 Not only are the strategies and methodologies of teachers trained in the ESP program consistent with the curriculum proposed by the MINERD, but the MINERD developed its curriculum with guidance from the CETT and ESP programs. Findings 4 and 5 relate to the perception of teachers and directors and changes in student learning, arguably the most important part of the ESP program. Consistently, teachers and directors, reflect that students are motivated and interested to learn reading and writing. They also suggest that the students are more interactive, engaged and responsible. They go on to note that this level of participation, which they attribute to the new methodologies, leads to improved student learning in reading and writing. Statistical analysis of assessment data examines if there are statistically significant differences in student learning in literacy (see Annex B). According to the analysis of students in the ESP, the average response rate correct on assessments for students in fourth grade has increased significantly each year between 2008 and 2011; the same is true in first grade between 2010 and 2011. However there was no significant increase in second or third grade (See Table 12, Annex B). When examining results from a control sample collected in 2010 and 2011, there are significant differences between intervention and non￾intervention schools in third and fourth grade, such that students in ESP schools on average perform better on the assessments3 (see Table 27, Annex B). At the same time, directors and teachers provide useful insights about skills which are more difficult to assess through standardized assessments. Teachers and directors perceive that students learn how to think for themselves, analyze, infer, and communicate effectively. As one Director stated, “the work with materials motivates them [the students], they think and innovate, they are able to think for themselves, not only what is in the book.” Based on these findings we can conclude that there is statistical evidence to support the perceptions held by teachers and school administrators that student learning has strengthened due to the ESP. Output 2 (cont): The pedagogical and methodological strategies of teachers trained are consistent with the curriculum proposed by the MINERD and applied in the classroom for strengthening student learning. Finding # 6: There is variation in strategies employed; teachers develop materials and infrequently dictate to students. Conclusion #4: Teachers are in the process of applying the strategies and methodologies in the classroom. The final group of findings related to Output 2 examine if the pedagogical changes teachers and directors describe can be observed taking place in the classroom. Additional details on teaching methodologies employed are included in Appendix A-5, which specifically examines the 14 classroom observations conducted during Spanish Language period with teachers who had either completed or were in the literacy intervention. Classroom observations indicated: • Materials on classroom walls: Teachers develop materials and they are seen in the classroom. Of all the classrooms observed (n=42 total observations), 50 percent had relevant materials on the walls that were age appropriate and were created by the teacher and students. An additional 32 percent had at least some materials (for a total of 82 percent). In some cases teachers went above and beyond; see the picture below. 3 Treatment and control groups were not randomly assigned. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 5 • Classroom Learning Corners4 : Forty-six percent of the classrooms observed had at least one Learning Corner. However, they varied in their development from simply a poster on the wall which said “Corner” to a developed resource about a specific theme. • Teaching strategies employed: None of the teachers were observed dictating to students. The most common pedagogical methods observed included teachers asking students questions out loud as a group (57 percent), followed by teachers writing questions on the board for students to answer (50 percent), and having students copy from the board (50 percent). Having students copy from the board is not reflective of the constructivist approach. 5 • Demonstrating with materials: Despite the fact that teachers and directors discussed students motivated to learn through the use of materials, only 15 percent of teachers demonstrated with materials. Based on these observations, it appears teachers are in the process of applying the strategies and methodologies in the classroom. Some strategies, for example designing classroom walls, are easier to implement than others, like teaching first graders to read using the letters of their names, see below. 4 Learning Corners are created by teachers and students with materials facilitated by the ESP and their own materials. They are a resource in the classroom about a specific content area and reflect the current curriculum being taught. Students should be able to access the Corners, read and engage with the materials. Although labeled “Corner” they are usually located on the classroom wall. 5 Constructivist teaching is based on the belief that learning occurs as learners are actively involved in a process of meaning and knowledge construction as opposed to passively receiving information. Constructivist teaching fosters critical thinking, and creates motivated and independent learners. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 6 What’s more, resource limitations (i.e. no workbooks) contribute to students spending large portions of time copying from the blackboard. (See Recommendations.) With regard to Output 3, “Competencies for critical reading and writing of first- fourth grade students improved” see discussion presented above (Finding 4) and Annex B for detailed statistical analysis. Output 4 states, “Teachers, directors, technicians, instructors and the program’s technical team document the experiences to propagate the model and make it sustainable.” Although there are various quarterly reports and other project documentation, the evaluation team did not receive any documentation, such as a systematized document of experiences, about the Literacy subcomponent or the ESP in general. (See Recommendations.) 2.1.2 In-service mathematics teacher training and curriculum development Output 1: Teachers’ mathematics competencies (knowledge, skills, aptitude and attitude) improved. Output 2: Teachers’ correct knowledge on mathematics curriculum and cross cutting themes included in the curriculum improved. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 7 Finding #7: Teachers participate in trainings and accompaniment and attribute these activities to growth in their mathematic competencies. Finding #8: Teachers have low capacity in terms of content knowledge of the mathematics curriculum; no subsequent assessment to detect improvement. Conclusion #1: Classroom level support visits are important to developing teacher competencies. Conclusion #5: The project has not provided sufficient evidence to determine if teacher capacity has improved, however it is an important issue to address given the low levels of teacher capacity evidenced. The Mathematics subcomponent has similar features to the Literacy subcomponent. Output 1 is virtually the same and Finding 7 is parallel to Finding 1 for the Literacy subcomponent and leads to the same conclusion. (Conclusion 1 is repeated in the above box.) Classroom level support, through the accompaniment visits is critical to developing teacher competencies. However, Output 2 specifically states: “Teachers’ correct knowledge of the mathematics curriculum and cross cutting themes included in the curriculum are improved;” whereas the Literacy subcomponent does not address teacher content knowledge. Key informant interviews revealed that, to date, it appears that teachers have low capacity in terms of content knowledge of the mathematics curriculum. Findings from an anonymous assessment administered to a sample of teachers who participate in the mathematics Saturday trainings seemed to yield similar findings. Related documentation on the “State of Numeracy in Latin America” indicates: “school teachers exhibit extraordinary weaknesses in their knowledge of content (only about half of the fourth-grade teachers in the provinces of Santiago and Santo Domingo recognized that the common fraction 1/2 is greater than 1/3).” (p. 12, Valverde & Naslund-Hadley, 2010). However, since the assessment was only administered once, the results provide no indication of whether or not there has been a change in teacher content knowledge. Therefore, the project has not provided sufficient evidence to determine if teacher capacity has improved. This is an important issue to address given the low levels of teacher capacity. (See Recommendations.) Output 3: Teachers’ skills to develop an annual teaching program that can be implemented in the time available in the school year are improved. Finding #9: Teachers have improved their planning processes but cannot cover all of the content in the mathematics curriculum. Conclusion #6: Teachers have improved their planning of the annual teaching program; however deficiencies (content knowledge) remain in implementation of the overall program. Teachers have improved their planning processes in order to implement the annual teaching program. This finding is reflected by school directors. As one director noted; “now teachers have plans; this is the requirement of the ESP, and the accompaniers help the teachers with their plans.” However, it also appears that teachers struggle to cover all of the content areas in the mathematics curriculum during the school year. For example, project data indicates that only 61.72 percent of teachers in third grade were able to cover the content corresponding to geometric figures and only 70 percent of teachers in fourth￾grade were able to cover the content corresponding to geometric figures (PEF, 2011). Based on these findings, it appears that teachers have improved their planning of the annual teaching program but that implementation of the overall program remains a challenge. There may be a relationship between the findings related to teacher capacity identified above (Finding 8) and ability to implement the annual teaching program. Teachers can only implement the program if they have the content knowledge in the areas (for example geometry) to do so. (See Recommendations.) AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 8 Output 4: Lesson plans are structured coherently in alignment with the MINERD’s curriculum and show the efficient time management programming. Finding #10: The ESP program is aligned to the MINERD’s mathematics curriculum; the ESP mathematics is the curriculum in the classroom. Finding #11: Teachers develop daily lesson plans reflective of the efficient time management programming but face challenges in implementation. Conclusion #7: Teachers create lesson plans based on the efficient time management program; they require reinforcement to ensure that plans are implemented as designed. Similar to the findings under the Literacy subcomponent, the Mathematics subcomponent is aligned with the MINERD curriculum. In fact, in reality the ESP is the curriculum in the classroom given that the MINERD requested the ESP to be implemented in all schools in the Northern region. Key informant interviews and project documentation indicate that the MINERD authorized the use of its own funds to purchase the mathematics texts for the Northern regions. In addition to annual plans, teachers develop daily lesson plans and they reflected the efficient time management program. Seventy-five percent of teachers observed had developed a lesson plan; with variation in terms of if they relied on them heavily, referred occasionally or did not use it at all. (See Appendix A-6). The lessons reflect an awareness of the efficient approach (building on knowledge, constructing knowledge, reviewing what was learned, etc). Students were also knowledgeable about the approach. Despite the plans, during classroom observations it appeared that teachers struggled to implement their plans in the time allocated; this appears due to (a) varying student learning needs, (b) large class sizes, (c) class discipline problems, and in some cases (d) teachers did not have sufficient command of the textbook (they would have the students complete all questions rather than selecting a representative few). These findings indicate that teachers create lesson plans based on the efficient time management approach but they do not always use them or not able to implement them as designed. (See Recommendations in terms of managing textbooks.) Output 5: Teachers use of strategies and adequate innovative materials are present in the mathematics teaching and learning process. Finding #12: Teachers and directors identify innovative materials as important to motivate learning in mathematics, particularly textbooks; they rely heavily on the textbooks and infrequently use other materials. Finding #13: Classroom observations indicate that, to a large extent, students are excited to learn math. Conclusion #8: Teachers value the materials and strategies but in application they use a mixed traditionalist/ constructivist approach. Conclusion #9: The mathematics textbooks are a fundamental ingredient to the teaching of mathematics. The analysis of classroom implementation continues with Output 5. Similar to the Literacy subcomponent, teachers and directors reflect on the use of materials to motivate student learning through a constructivist approach. Teachers and directors frequently referred the ESP mathematics materials and their importance. In interviews they discussed the mathematics textbooks and the mathematics box materials including the geometric figures, the blocks of ten, among others. Classroom observations confirm that students are motivated to learn mathematics; the fear of mathematics has been dissipated. As one teacher noted, “before there was a fear of mathematics, now it is improved, the kids are interested in AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 9 mathematics.” See Appendix A-6 for the 21 classroom observations conducted during the mathematics period with teachers who had either completed or were in the mathematics intervention. Classroom observations indicated: • Mathematics textbooks: Teachers rely on the mathematics textbooks, in classroom observations during mathematics classes (n=21), nearly 70 percent of the observations indicated that all students used textbooks; the exception was in Herrera (1505). • Using teaching-learning materials: While teachers referenced the ESP mathematics materials during interviews, observations indicated only 8 percent of students and 25 percent of teachers used materials during classroom observations. • Teaching strategies employed: The strategies employed by teachers were largely: teacher writes on board to: have students read (70 percent), answer questions (58 percent), and for discussion (52 percent). Teachers appear to believe in the constructivist strategies but in practice rely on a mix of traditional and constructivist approaches. This may be due to teacher capacity or, as previously indicated, other challenges in the classroom. The structured constructivist approach employed may be the most effective approach given the limitations teachers face. (See Recommendations.) What’s more, the mathematics textbooks are a fundamental ingredient for learning in the classroom. As one teacher indicated, “the books apply the steps and students can work independently.” In terms of Output 6, “Teachers’ competencies to develop student evaluation strategies improved” despite the fact that the MINERD requires formative/ summative evaluation; it did not appear to be a focus of teachers’ work. For a discussion on results in mathematics performance see the Monitoring and AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 10 Evaluation component (Section 2.4) and detailed analysis in Annex B; note the project does not define changes in student learning as an output of the Mathematics subcomponent. 2.2 Education Management Component The Education Management Component piloted the “Quality Management” model developed by the MINERD, with the objective of strengthening school governance by working with district directors, technicians, school directors and parents’ associations in four districts (1503, 1505, 0803, 1603). There are three Outputs identified in project documentation. Findings are predominately related to the first two outputs. Output 1: Improve the capacity to perform the functions of technical assistance, mentoring, supervision and evaluation by district technical staff and school directors. (The output is best examined as it applies to school directors and then district technical staff.) Output 2: Implement Education Management Component in centers for education management, institutional management and creating learning communities. Finding #14: School directors associate the trainings they receive with their ability to perform their roles. Finding #15: School directors infrequently identify school management support from district technical staff. Finding #16: District technical staff who have participated in the program are knowledgeable about the “Quality Management” model. Conclusion #10: The direct intervention approach of trainings to school directors is more effective than the cascade model via district technicians. Conclusion #11: Given the administrative demands on directors and on districts; it is more effective for project technical staff to provide accompaniment directly to directors. School directors reflect on the trainings they receive and associate them with their ability to perform their roles as directors. In particular, they identify the “Director’s Diploma” as improving their ability to lead, manage, and communicate with their schools and school communities. The ESP is not responsible for the “Director’s School,” a MINERD lead initiative; however the ESP offered the module on leadership to directors who participate in the “Director’s School.” Directors also reflect positively on participation in this program. School directors infrequently identify district technical staff as supporting them in the management of their schools. It is important to note that this was consistent in districts in which the Education Management intervenes, which is designed to build the capacity of districts to support school directors. Changes at the central level of the Minister (February, 2011) and related policies regarding decentralization may complicate findings related to the effectiveness of the district technical staff, however this reality further reinforces the importance of working at the school base, rather than a trickle down approach which can be plagued by political turnover. However, the reality is that school directors face numerous administrative responsibilities (and cannot leave school for trainings). In many cases, if there is a coordinator or sub director, they take on the pedagogical leadership roles. Similarly, district technical staff who participate in the Education Management component have multiple additional responsibilities. For example, one key informant interview with a district technician indicated that she supported over 50 schools in addition to the schools she worked with for the Education Management component. While district technicians in the program AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 11 know the “Quality Management” model and its components, they are not effectively transferring these components to their directors. In light of administrative demands on school directors it is most effective to bring reinforcement to the school center, via accompaniment by project technical staff. (For more discussion on creating learning communities, especially for/with parents, see Sustainability.) In terms of the final Output 3, “The districts and centers have a monitoring and evaluation system that allows for the “Quality Management” model to verify results,” key informant interviews with districts and central MINERD verified that a process is in place and referenced the “System of Accompaniment and Supervision,” however no profound findings or conclusions emerged. 2.3 Safe Schools program The focus of the Safe Schools intervention is to integrate students, teachers, parents and community members in the process of addressing School-Related Gender Based Violence (SRGBV). The program was piloted in two districts Herrera, Santo Domingo (1505) and Jarabacoa, La Vega (0603) in a total of 41 schools. Output 1,2,3: Help students (Output 1) teachers (Output 2) and communities (Output 3) learn how to prevent violence and increase their self-efficacy through enhanced knowledge, attitudes and skills regarding healthy relationships, reproductive health issues, HIV prevention and children’s rights and responsibilities. Finding #17: Youth discuss with ease the rights of the child, they are most confident and comfortable discussing this area of the program; they are not as familiar or do not feel comfortable discussing other areas addressed in the Safe Schools curriculum. Finding #18: Teachers, parents, directors identify discipline and violence problems in schools, homes and communities; they often attribute these problems with challenges in student leaning. Conclusion #12: Youth discuss themes related to the rights of the child with ease; sensitive themes in the Safe Schools curriculum require time, trust, and the appropriate cultural context to develop. Conclusion #13: The Safe Schools program, through its broad based approach to community involvement, has the potential to influence student learning through addressing discipline and violence problems that plague homes, communities and schools. In group interviews with youth participants in the Safe Schools program, it appears that youth easily discuss the rights of the child and they are most confident and comfortable discussing this area of the program. For example, one youth expressed, “the rights of the child are to study, eat, have a roof over your head, to have a family and to have love.” Another articulated, “children have the right to not be treated badly and to express their opinions.” However, youth are not as familiar or do not feel comfortable discussing other areas addressed in the Safe Schools curriculum, for example, SRGBV. It appears that the sensitive themes in the Safe Schools curriculum require time, trust, and the appropriate cultural context to develop. With regard to parents, teachers, and directors, these actors consistently identify discipline and violence problems in schools, homes and communities. They often attribute these problems with challenges in student leaning. The Safe Schools program, through its broad based approach to community involvement, has the potential to influence student learning through addressing discipline and violence problems that plague homes, communities and schools. (See section on Efficiency with regard to project AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 12 coordination.) It should be noted that to date, the Doorways manuals, have been culturally adapted but final reproductions have not been distributed. 2.4 Monitoring and Evaluation The Monitoring and Evaluation component is designed to “assess the impact of the different program components on students’ learning… and to provide valid and reliable information for feedback and refinement of program processes.” (ESP, 2012b). It also is designed to provide the MINERD with information to make decisions. With regard to Output 1, “Evaluation of student learning in mathematics and reading comprehension,” see detailed Annex B. Findings with regard to performance in reading comprehension were identified in Section 2.1.1 given the literacy subcomponent specifically identifies student learning improvement as an output. On the contrary, the mathematics subcomponent does not identify student learning as an output, it only identifies teacher content knowledge improvement (which the literacy subcomponent does not identify). However, it is important to note with regard to mathematics that the statistical analysis indicates the following: of students in the ESP, on average responses correct on student assessments increased significantly each year between 2008 and 2011 in fourth grade and between 2008 and 2010 in third grade; however there were no significant increases in average responses correct in first or second grade between 2010 and 2011 (see Table 19, Annex B. Comparison data indicates that on average students in ESP schools performed better on mathematics assessments than their control group peers in third and fourth grade in 2010 and 20116 (see Table 27, Annex B). In general terms student learning in mathematics has strengthened, with exceptions. Output 2: Evaluation of teacher knowledge of mathematics and reading comprehension. Finding #19: There was one anonymous evaluation of teachers in mathematics conducted with a sample of teachers who participate in the Saturday mathematics program, using a test for students and no evaluation of teachers in reading comprehension. Conclusion #14: There has been no rigorous evaluation of teacher knowledge of mathematics or reading comprehension to date to detect change. With regard to Output 2 “Evaluation of teacher knowledge of mathematics and reading comprehension,” discussion and findings related to mathematics can be found in section 2.1.1; there has been no evaluation to date in literacy. Output 3, “Evaluation of school governance practices,” pertains to evaluation within the Education Management component. This evaluation has not been conducted, but key informant interviews suggest that three assessments exist which will be utilized in achieving this output. These assessments are the Segundo Estudio Regional Comparativo y Explicativo (SERCE) survey for principals, the Research Triangle Institute (RTI) Principal Survey, and a previously developed survey from mathematics and literacy components for principals. While no profound findings emerge, the component may want to consider including other participants who are beneficiaries of the Education Management component in the evaluation. Output 4: Assist the MINERD in developing capacity building plans in terms of evaluation, testing, measurement, and analysis skills for the General Direction of Evaluation. 6 Treatment and control groups were not randomly assigned. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 13 Finding #20: Currently, the Direction of Evaluation is conducting three assessments simultaneously. Finding #21: There are a limited number of professionals with the skills required to conduct evaluation work. Finding #22: INOFOCAM is developing a scholarship program with input from the ESP evaluation component for university students to study testing and evaluation. Conclusion #15: The MINERD has the capacity to build plans for evaluation, testing, measurement and analysis; the challenge is having enough human capital resources to implement them simultaneously. Currently the Direction of Evaluation is conducting three assessments simultaneously; international assessments (UNESCO), assessments for graduation, and assessments with a sample of schools to compare the three interventions (ESP, OEI, and POVEDA). These multiple demands are stretching the Direction of Evaluation’s limited resources. There are a limited number of professionals with the skills required to conduct evaluation work in the DR. In order to address this challenge, key informant interviews revealed that INOFOCAM is developing a scholarship program with input from the ESP Monitoring and Evaluation component for university students to study testing and evaluation. It appears that the MINERD has the capacity to build plans for evaluation, testing, measurement and analysis. The challenge is having enough human capital resources to implement them simultaneously. 3. Efficiency Based on findings and conclusions related to performance, efficiency must be examined. Efficiency pertains to the relationship between inputs and outputs; this evaluation employs the definition: “The utilization of USAID investment and effort in design, management and execution of program components” (AMEX International, 2012). Guiding questions include: What are the aspects of the projects that are operating more or less efficiently and why?, What are the necessary and sufficient conditions for efficient implementation?, and How can the design, management and execution be more efficient in achieving program goals? First findings related to project management are identified. Then findings related to implementation in schools are discussed. Project Efficiency Finding #23: Working autonomously allows each component to adjust to specific needs and realize efficiency gains, however, there opportunities for mutual reinforcement and interdependence between the project components that is not expressed. Conclusion #16: There are tradeoffs in working independently; efficiency or outputs are not maximized to their potential from project collaboration. As a program composed of different components, it appears that each component operates autonomously which allows the components to be adjusted to specific needs encountered in the field. For example, components can adapt training programs or adjust to changes in ministry policies. At the same time it also appears that there are commonalities across components at the project and school level. For example; (1) literacy, mathematics, education management, and Safe Schools all employ a methodology reflective of “constructivism;” (2) the literacy and mathematics interventions work in the same schools, at times teachers receive accompaniment from two project staff at the same time. However, shared aspects of components do not support one another. For example, (1) the evaluation information from the Monitoring and Evaluation component could be disseminated via the Education Management component to parents, teachers, and directors, (2) Education Management’s work with parents does not broadly AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 14 support literacy or mathematics interventions (literacy fairs and mathematics Olympics which include parental participation do not coordinate with the Education Management component). With regard to the Mathematics and Literacy subcomponents there is interdependence between the two that is not expressed. For example, classroom observations indicate that: (1) teachers use mathematics textbooks to teach their Spanish Language classes, in other words, in order to teach their Spanish Language classes with limited resources, they rely on the mathematics textbooks for access to a text to teach reading and writing literacy skills, and (2) students in advanced grades can only work independently in mathematics books if they can read. Along these lines, when asked: “Do students respond to the new methodologies in mathematics?” a member of the mathematics technical team answered: “There is a problem, with reading, in order to solve mathematics problems you have to be able to read.” The relationships between the components are important to consider as they pertain to efficiency, especially with regard to literacy and mathematics and USAID’s strategy (see Final Thoughts). Working autonomously allows each component to adjust to specific needs and realize efficiency gains. However, it also appears that there are tradeoffs in working in isolation. In turn, efficiency or outputs are not maximized to their potential from project collaboration. Project Efficiency (cont) Finding #24: The ESP is systematic, organized, and efficient, as perceived by the private sector and the MINERD. Finding #25: There are challenges in accessing reliable and accurate administrative information about the program. Conclusion #17: The ESP administration is efficient, but can be improved. During key informant interviews with the private sector and the MINERD, interviewees consistently noted that the ESP is systematic, organized, and efficient. As one member of the private sector indicated, “The ESP is distinct with their level of efficiency, quality of the materials, and the technical quality of their team.” A member of the MINERD noted with regard to evaluation, “The ESP is the project that has everything systematized and organized, they demonstrate year after year what they are doing.” This was also the experience of the evaluation team, with the exception of challenges experienced in accessing program administrative information, for example, how many schools are in the program. This was seen in the most recent report document (ESP, 2012a) which suggests that there are: 89 schools in Spanish Language, 192 schools in Mathematics, 225 schools in Education Management and 41 schools in Safe Schools, for a total of 547 schools. However this does not reflect the fact that these various components are often implemented in the same schools. Implementation Efficiency Finding #26: There is a perception that materials are costly, however each time materials are reproduced the overall investment increases (cost per book decreases) because the materials have already been developed and validated. Finding #27: Materials have been purchased that have not been distributed to schools or are underutilized in schools. Conclusion #18: To ensure the efficiency of materials they must be distributed and used in classrooms. Conclusion #19: There are efficiency loses AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 15 Finding #28: Teachers leave/ rotate schools frequently. Finding #29: Teachers teach in more than one session (i.e. morning and afternoon). Finding #30: Teachers in pre-service teacher training programs have practicum experiences in the ESP schools and learn strategies from teachers in the program. when teachers leave schools, at the same time the ESP may be under representing some efficiency gains with regard to the benefits of teachers trained in their program. There is a perception that materials are costly, as reflected in key informant interviews with the private sector and with the MINERD. However it is also clear that each time the investment in materials decreases because the materials have already been developed and validated in DR classrooms. On average, mathematics textbooks cost $2.00 USD. However, regardless of views on initial cost, it is crucial to note that materials have been purchased that have not been distributed to schools or are underutilized in schools. For example, the classroom libraries encountered distribution complications and during classroom observations were not found in schools, similarly mathematics box materials were underutilized, see Appendices E and F respectively. Each day that goes by that students do not have access to materials is a depreciating rate of return on the initial investment. To ensure the efficiency of materials they must be distributed and used in classrooms. Similar to the investment made in materials is the investment made in training teachers. Teachers leave their teaching positions frequently. When they go to other schools, they may bring with them the methodologies they have learned from the ESP, although given findings on accompaniment and a culture of support, it is unlikely that the unaccounted positive spillovers are greater than the losses in efficiency. At the same time, positive gains arise when teachers teach in two sessions (morning and afternoon). For example, of available data from teacher interviews over 63 percent of teachers taught in more than one session (examined 22 interviews, 12 indicated they teach more than one session and 3 did not indicate a response). In some cases teachers taught in a grade other than the first cycle. What’s more, during school visits, it was noted that student teachers complete their practicums or student teaching experiences with teachers from the ESP who share with the new teachers the ESP strategies. As such, while there are efficiency loses when teachers leave schools, there may be some under representation of efficiency gains. 4. Sustainability With a foundation of findings and conclusions on the ESP’s performance and efficiency the discussion can turn towards sustainability or “the likelihood that components will continue without USAID assistance;” AMEX International (2012). The guiding questions include: Which program components are sustainable (most/ least- why)?, Does the private sector/ GoDR believe the program is important enough to be supported?, and Do program beneficiaries believe that the benefits are important enough to continue to seek services? Sustainability is viewed first in terms of if behavioral changes will sustain and then in terms of the continuation of the program. The link between the two is made. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 16 Sustaining Behaviors Finding #31: Teachers participate in a professional program that becomes a part of their professional career trajectory. Finding #32: Teachers are convinced of the learning benefits in the methodologies and are committed to implementing them; they continue to implement them even after training is complete. Finding #33: Teachers feel isolated in schools where they do not have a support community of fellow teachers who implement the same approach. Finding #34: Teachers depend on materials, for example the mathematics textbooks, in order to be able to implement the strategies. Conclusion #20: Teachers are committed to sustaining the interventions in literacy and mathematics and they do; especially when they have a community of support. Conclusion #21: Teachers will need material resources, for example textbooks, in order to sustain the approach. This section looks at the sustainability of changes in behaviors, particularly of those of teachers who are the main focus of the ESP program. Teachers participate in a professional program implemented by a highly esteemed university in the DR; it becomes a part of their professional career trajectory. Key informant interviews and teachers confirm that the program is different from a sporadic training, and is recognized with either a certificate or a diploma which are acknowledged in the national teachers’ competitions for teaching positions. What’s more, teachers are convinced of learning benefits of the ESP methodologies (in both subject areas) and are committed to implementing them. Teachers appear to continue to implement ESP strategies even when the interventions are complete. For example, in school 03856, the teacher had completed the training program and continued to implement the strategies in the classroom. This same teacher left the school and went to a non-ESP school, when he did it was challenging to continue to implement strategies. Teachers feel isolated in schools where they do not have support community of fellow teachers and a director who implement the same approach. Teachers are committed to sustaining the interventions in literacy and mathematics and they do; especially when they have a community of support. Similar to school culture for sustaining behaviors and as discussed above, teachers depend on crucial materials, for example the mathematics textbooks, in order to continue to implement strategies. It is virtually impossible to continue to implement the mathematics methodology without this resource and teachers will not be able to sustain the approach without it. Program Sustainability: The Ministry of Education (MINERD) Finding #35: The MINERD has requested the expansion of the program and uses its own resources coupled with donor funds to expand the program in the Northern region. Finding #36: The MINERD has limited funds for expansion activities and the intervention has not been implemented with the same design of the full ESP program. Conclusion #22: Aspects of the ESP program can be sustained; however tradeoffs will be made resulting in sacrifices in terms of quality. How will classroom visits continue without project technical staff? Who will be responsible for providing mathematics textbooks? In order to answer these questions the analysis turns towards program AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 17 sustainability and the role of three sets of actors: (1) the state, in this case the GoDR and specifically the MINERD (central and decentralized levels), (2) the private sector and (3) communities, in particular, parents. As noted earlier, districts are not able to implement the approach due to their limited capacity. It seems that aspects of the ESP program can be sustained, but tradeoffs will be made resulting in sacrifices in quality. The MINERD has requested the expansion of the program and uses its own resources coupled with donor funds to expand the program in the Northern region. PUCMM receives funds from the MINERD and has shifted its own resources in order to implement the intervention in expansion schools (ESP-MINERD schools) via a lighter approach which includes training school coordinators. However, the MINERD has limited funds for expansion activities and the intervention has not been implemented with the same design as the full ESP program. Program Sustainability: Donors and Lending Agencies Finding #37: IDB funds have been used to support expansion of the ESP program however there are concerns about funding programs without evaluation; future funding will focus on infrastructure. Conclusion #23: In order to garner resources programs including ESP need to conduct rigorous evaluations and disseminate findings. Conclusion #24: It is unlikely funds to support ESP will come from IDB. MINERD will need to use its own resources. Given the MINERD’s resource constraints, it is important to consider the role of donors and lending agencies. The largest donor in the DR is the Inter-American Development Bank (IDB). IDB funds have been used to support the expansion (including: ESP, POVEDA and OEI), however the funds are limited. Key informant interviews indicate that only $US 3.5 Million of the original $US 7 Million funds have been allocated for the expansion. Specifically, the IDB is funding an evaluation of the programs. There is apprehension about funding programs on a national scale that have not been rigorously evaluated. In order to garner resources, programs (including the ESP) need to conduct rigorous evaluations and disseminate findings. Lastly, the new IDB loan for $US 50 Million is primarily for infrastructure with 15 percent for the extended class schedule. It is unlikely that funds will be used for basic education; this implies that funds for sustaining ESP activities by MINERD will need to come from sources other than the IDB. Program Sustainability: The Private Sector Finding #38: Fundación INICIA supported the ESP because they were presented with results and were convinced by the program’s methods; schools were selected where MINERD or USAID programs are not already being implemented. Conclusion #25: The private sector can be included in the expansion of the program through convincing them of the program and its results. Conclusion #26: The relationship can be a partnership between PUCMM, the private sector actor, and the MINERD in order to maximize results. It is also important to consider the private sector when discussing sustainability. Fundación INICIA supported the ESP because they were presented with results and were convinced by the program’s methods. To date Fundación INICIA has invested over $US 1.5 Million to expand the program in a select number of schools (49 schools) in the Cibao region. These ESP-INICIA schools were selected following the negotiations to determine the ESP-USAID schools and ESP-MINERD schools. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 18 The private sector can be included in the expansion of the program through convincing them of the program and its results. In this case, the partnership between PUCMM, the private sector actor, and the MINERD ensures the greatest number of schools receive basic education support in the Northern region of the DR, rather than replicating interventions in the same schools. It is also important to note that, in this case, Fundación INICIA has accepted the full ESP program in mathematics (and will do the same in literacy) and implements the program as designed (including all materials, trainings, and visits). However, Fundación INICIA also has the liberty of only selecting 49 schools with which to work; compared to the MINERD. These conclusions have implications beyond just the ESP program; see the Evaluation Portfolio Evaluation Integrated Report. Program Sustainability: Communities and Parents Finding #39: Teachers and directors argue that lack of parental participation is the source of school problems and prevents achieving learning goals. Finding #40: Parents participate in their children’s education at home and at school; when they do not participate it is because they do not know how or face competing demands. Conclusion #27: There is a disconnect between the perception of teachers, directors and parents about parents’ role in their children’s education. Conclusion #28: Parents want to help their children succeed in school; they require a supportive school community context and training to be able to do so. School communities and, in particular, parents are a fundamental pillar of sustainability. During interviews teachers and directors consistently argue that lack of parental participation is the source for school problems and a challenge for achieving mathematic and literacy goals. At the same time, interviews with parents suggest that in many cases parents do participate in their children’s education, through in school and at home activities. Interviews also indicate that parents are resourceful in finding ways to help their children, especially when they may not have the specific skills to do so, for example one mother noted, “I am illiterate so I send my daughter to my sister to help her with her homework.” In other cases, parents do not participate in their children’s education because they do not know how or because they face competing work or home demands, such as care for another child. As one parent indicated, “sometimes we are mother and father at the same time.” These findings suggest that there is a disconnect between the perception of teachers, directors and parents about how parents participate in their children’s education. The conclusion also emerges that parents want to help their children succeed in school; however, they require a supportive school community context and training to be able to do so. (See Recommendations.) 5. Recommendations This section draws together the findings and conclusions from performance, efficiency and sustainability into six coherent recommendations for the duration of the project and one recommendation for future consideration, they include: AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 19 Recommendation #1: Continue to implement the program as designed and exercise caution in shifting resources for expansion demands; in particular maintain the classroom visits to teachers and school visits to directors. Based on the Conclusions: Conclusion #1: Classroom level support visits are important to developing teacher competencies. Conclusion #4: Teachers are in the process of applying the strategies and methodologies in the classroom. Conclusion #6: Teachers have improved their planning of the annual teaching program; however deficiencies remain in implementation of the overall program. Conclusion #7: Teachers create lesson plans based on the efficient time management program; they require reinforcement to ensure that plans are implemented as designed. Conclusion #11: Given the administrative demands on directors and on districts; it is more effective for project technical staff to provide accompaniment directly to directors. Recommendation #2: Ensure that materials are reproduced and delivered and trainings are offered focusing on their use, especially the mathematics textbooks, literacy workbooks, and classroom libraries. Based on the Conclusions: Conclusion #8: Teachers value the materials and strategies but in application they use a mixed traditionalist/ constructivist approach. Conclusion #18: To ensure the efficiency of materials they must be distributed and used in classrooms. Conclusion #21: Teachers will need material resources, for example textbooks, in order to sustain the approach. Recommendation #3: Create a user friendly systematized document of the ESP experience with project components and findings from evaluations and studies. Disseminate the product. Based on the Conclusions: Conclusion #23: In order to garner resources programs including ESP need to conduct rigorous evaluations and disseminate findings. Conclusion #25: The private sector can be included in the expansion of the program through convincing them of the program and its results. Recommendation #4: The ESP should seek opportunities to collaborate across components at the project and at the school level. Based on the Conclusions: Conclusion #16: There are trade-offs in working independently; efficiency or outputs are not maximized to their potential from project collaboration. Conclusion #17: The ESP administration is efficient, but can be improved. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 20 Recommendation #5: Concentrate project activities directly with parents and directors at the school level to sustain ESP interventions and achievements; consider expanding Mothers and Fathers in Action7 . Based on the Conclusions: Conclusion #10: The direct intervention approach of trainings to school directors is more effective than the cascade model via district technicians. Conclusion #11: Given the administrative demands on directors and on districts; it is more effective for project technical staff to provide accompaniment directly to directors. Conclusion #27: There is a disconnect between the perception of teachers, directors and parents about parents’ role in their children’s education. Conclusion #28: Parents want to help their children succeed in school; they require a supportive school community context and training to be able to do so. Recommendation #6: Ensure student evaluations and teacher evaluations are implemented, analyzed and results disseminated to all levels of the education system to inform decision making; hire someone locally to respond to ongoing M&E needs. Based on the Conclusions: Conclusion #14: There has been no rigorous evaluation of teacher knowledge of mathematics or reading comprehension to date to detect change. Conclusion #17: The ESP administration is efficient, but can be improved. Conclusion #23: In order to garner resources programs including ESP need to conduct rigorous evaluations and disseminate findings For future consideration… Recommendation #1: Reinforce pre-service teacher training program to address the root of low teacher capacity; design a pre-service ESP pilot program for a small sample of teachers for long-term systemic change; and evaluate the change in teacher content knowledge. Based on the Conclusions: Conclusion #5: The project has not provided sufficient evidence to determine if teacher capacity has improved, however it is an important issue to address given the low levels of teacher capacity evidenced. Conclusion #6: Teachers have improved their planning of the annual teaching program; however deficiencies (content knowledge) remain in implementation of the overall program. 6. Final Thoughts In conclusion, this evaluation has provided insight on performance, sustainability and efficiency of the ESP program. In light of the USAID Strategy (2011-2015) it is important to recall that CETT focused on literacy, TEF focused on mathematics and ESP focuses on both. In the second half of the ESP program it will be essential to reflect on the Mathematics and Literacy subcomponents and their interdependence. With a swing back towards literacy interventions reflected in USAID’s current Education Strategy, it may be pertinent to consider aspects of the Mathematics subcomponent, that despite being a different content area, may apply to reaching literacy goals. For example, a more structured constructivist approach, a concentrated training program on Saturdays, or a stand-alone resource, like workbooks available for each student, that can facilitate the teaching-learning process. 7 The Mothers and Fathers in Action program was piloted under the Education Management component in which parents received trainings on how to help their students at home. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 21 Appendix A -1. Map of Expansion, source: MINERD, 2011 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 22 Appendix A-2. Description of Interventions: ESP, POVEDA, OEI Table 2. Interventions in Mathematics ESP POVEDA OEI Strategies Strategies Strategies • Training “presencial” of 150 hours for teachers • Strategies for modeling mathematics learning • Plan for accompanying students in first grade that present special needs in mathematics • Accompaniment in the classroom, between 20- 30 hours each teacher • Strategies for learning mathematics from the DR curriculum • Pedagogical support in mathematics • Mathematics Olympics • Training “presencial” for 54 hours • Meet every fifteen days to identify themes with weaknesses • Clubs for mathematics and science • 16 hours of accompaniment • Review special cases that present difficulty • Strategies for parents to help with their children • Space for action￾reflection in the learning community • Teachers reflect on their own practice Table 1. Interventions in Literacy ESP POVEDA OEI Strategies Strategies Strategies • Training “presencial” 24 hours, annually, three times a year • Diversity of text in the classroom • A plan of accompaniment based on the special needs in reading and writing • Innovation Circles • A focus on text, function and communication of language • Support in reading and writing • Independent study 40 hours annually • Diversity in learning writing and reading • Reading and writing fairs by school networks • Accompaniment in the classroom, between 20- 30 hours each teacher • Strategies for acquiring and developing reading competencies and writing production • Literacy for mothers and attention to children younger than 6 years old • Annual fair for reading/ writing • A lettered environment • Competition for children stories, annually Resources for teachers • A book for training • Guides for reading￾writing • Interactive modules, videos, and audio resources • A lettered environment (materials) Resources for the student • Classroom library • Diagnostic and formative assessments AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 23 • Environmental themes • Individual reflection for 40 hours annually • Trainings that focus on human development and the development of the teacher as a person • Seminars for 4 hours every district • Training of teaching teams within the school • A training for each district for 16 hours, and accompaniment for each session per school for 4 hours • Accompaniment in the classroom with teacher pairs and pedagogical groups and school center networks • Use of technology to exchange with teachers in other countires • Elaborate trimester plans for monitoring by OEI • Communication with the authorities of MINERD Resources Resources • A book for each child 1-4 grade • Laboratory • A guide for teachers • Manipulative materials, audio visual materials, communication materials Source: ESP, 2010, 2011, 2012; MINERD, 2011 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 24 Appendix A-3. List of Key Informant Interviews Name Title Organization Date Liliana Degiorgis Investigator EDUCA 10/2/2012 Marina Taveras Agreements Officer Technical Representative USAID 10/3/2012 Kevin Roberts Education Specialist USAID 10/3/2012 Sarah González de Lora Vicerrectora de Investigación e Innovación Pontificia Universidad Católica Madre y Maestra 10/3/2012 Francisco G. Estrella F. Dir. General de Auditoria y Control de Procesos Educativos Ministerio de Educacion 10/3/2012 Ancell Scheker Mendoza Dir. General de Evaluacion y Control de la Calidad de la Educacion Ministerio de Educacion 10/4/2012 Jose Alberto Contreras (4 yrs), Tania Jimenez Rosa (4 yrs), Oscar Amargos Coordinadores de Processos￾Edu Mgt Pontificia Universidad Católica Madre y Maestra 10/5/2012 Romona Santana (4yrs) Asistente Tecnico de Padres y Madres en Accion Pontificia Universidad Católica Madre y Maestra 10/5/2012 Barbara Campos, Sonya Mendez Coordinadores de MAT Pontificia Universidad Católica Madre y Maestra 10/8/2012 Gineda Castillo previous Directora Ejucativa INOFOCAM, pending Dir. Edu Mgt *Pontificia Universidad Católica Madre y Maestra/ INOFOCAM 10/8/2012 Marys Encarnacion, Jose Anibal Paula Ceballos, Denia Feliz Gomez, Dulce Maria del Rosario Accompanantes de Gestion Pontificia Universidad Católica Madre y Maestra 10/8/2012 Catalina Andujar Schekler Representante Permanente de la OEI Organizacion de Estados Iberoamericanos 10/10/2012 students, female,Franciel Hernandez (12), Nicole Torro (13) Salesiana San Jose, Herrera (1505); Safe Schools 10/11/2012 Teacher, Angela Gonzalez Salesiana San Jose, Herrera (1505); Safe Schools 10/11/2012 Denia Burgos Directora Ejucativa INOFOCAM 10/12/2012 Sarah Guilamo Directora, Escuelas Seguras PUCMM 10/12/2012 Victoria Rosado (2 years), Daily Yajahira Perez Salvador (5 years), Ariel Pacheco Capacitadoras- LE PUCMM 10/12/2012 Alta Gracia Herrera (3 years) Coordinadora- LE [same interview as 014] PUCMM 10/12/2012 Angela Espanol Gerente de Projectos Fundacion Inicia 10/12/2012 Ana Dolores Guzman Directora Ejucativa ISFODOSA 10/12/2012 Gilbert Valverde Sr. Researcher, Academic Chair, Director of Evaluation Component SUNY-Albany 10/17/2012 Eleuterio Martinez Accompante Matematicas PUCMM 10/17/2012 Rosario Concepcion (late), Yajaira Munoz (6, 4, 3), Ramona Garcia (4), Gleiry Munoz (3, 1) Madres de le la escuela￾Mauricia Parello Rochet, Santiago 10/18/2012 Maria Veronica Rosario Tecnica del Distrito 0803 Distrito 0803 10/18/2012 Dra. Liliana Montenegro Directora del Centro de Excelencia para la Investigacion y Difusion de la Lectura y Escritura (CEDILE) 10/22/2012 Rosalina Aquio Ortiz (1,3, 5), Rosa Maria Lena Mora (4, 5, 5), Santa Monica Abreu (2) Madres de la escuela- Barrio Blanco, Jarabacoa 10/23/2012 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 25 Rosalba Adames (14), Luisa Maria (12), Nicol Bremont (13), Melizza Perez (12), Adonis Pena (13) Safe Schools jovenes- Barrio Blanco, Jarabacoa 10/23/2012 Willianna Jacqueline Abreau Molina (13), Britney Yelena Abreu Molina (10), Lissy Masiel Alonzo Abreu (13), Filneiry Belliard Urena (10) Safe Schools jovenes- Jose Antonio Guzman, Jarabacoa 10/23/2012 Fernando Ogando MINERD 10/25/2012 Angel Daniel Franco Madera Ecargado de Planificacion, tecnico del Distrito Distrito 1503 10/29/2012 Horacio Alvarez Marinelli Education Specialist Inter-American Development Bank 10/29/2012 Rita Ceballos, Onelda Gomez Rita*- Directora General, Onelda￾Coordinara del primer ciclo MINERD- Direcion de Educacion Basica; *coordinated POVEDA 10/29/2012 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 26 Appendix A-4. List of Schools visited Date School Code Tanda Director interview Teacher interview(s) Observation(s) 10/9 Anibal Ponce 00132 Ves 1 2 2 10/11 Salesiana San Jose 00421 Mat 1 1 2 10/11 Curazao Libertador 00114 Mat 1 2 2 10/11 Basica Herrera/ Cristobalina Batista 00015 Ves 1 1 1 10/11 Basica Durate 00010 Ves 1 1 1 10/15 Jose Antonio Cespedes 04399 Mat 1 1 2 10/15 Padre Betancourt 04405 Mat 1 1 2 10/15 Sebastian Paredes 04385 Ves 1 1 2 10/16 Federico Munoz, Prof. Caimito Afuera 01856 Ves 1 1 2 10/17 Bao 03973 Mat 1 1 3 10/17 Francisco Ramon Collado 04051 Ves 1 1 3 10/17 Francisco Antonio de las nueces 03924 Mat 1 1 1 10/17 Genaro Perez 03842 Mat 1 1 2 10/18 Marcelino Valenzuela 04175 Mat 1 1 2 10/18 Rafaela Jiminian de Cruz 04172 Mat 1 2 2 10/18 Mauricia Perello Rochet- la Cruz de Isalguez 03991 Ves 1 1 1 10/23 Barrio Blanco 01956 Mat 1 0 2 10/23 Piedra Blanca Hatillo 01975 Mat 1 1 1 10/23 Jose Antonio Guzman 01960 Ves 1 1 1 10/23 Ana Graciela Morillo 01819 Ves 1 2 2 10/24 Olivia Capella 02521 Ves 1 1 2 10/24 Sabana Grande 02551 Ves 1 1 2 10/24 Fermina Altagracia 03856 Mat 1 1 2 10/24 El Guano 04133 Ves 1 2 Total 24 28 42 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 27 Appendix A-5. Classroom observations (data related to Literacy subcomponent) Table 2. Learning Corners # % Have 18 46 Do not have 21 54 Missing 3 Total 42 Figure 1. Teaching-learning strategies observed, n= 14 (classroom observations- LE) Q13A1= WRITES ON BOARD FOR DISCUSSION *Q13A2= WRITES ON BOARD TO ANSWER QUESTIONS *Q13A3= WRITES ON BOARD TO HAVE STUDENTS COPY Q13A3= WRITES ON BOARD TO SUMMARIZE Q13A4= WRITES ON BOARD TO HAVE STUDENTS READ Q13B1= TEACHER DICTATES Q13B2=TEACHER READS ALOUD Q13B3= TEACHER ASKS STUDENTS QS INDV *Q13B4= TEACHER ASKS STUDENTS QS GROUP Q13C= TEACHER DEMONSTRTES PHYSICALLY Q13D= TEACHER DEMONSTRATES WITH MATERIALS Q13E= TEACHER DRAWS Q13F= TEACHER USES POSTER Q13G= STUDENTS READ ALOUD Q13H= STUDENTS READ SILENTLY Q13I= STUDENTS ROLE PLAY Q13K= STUDENTS DEMONSTRATE PHYSICALLY Q13L= STUDENTS DEMONSTRATE WITH MATERIALS *Q13M= STUDENTS DO SEATWORK Table 1. Education Material on Walls # % Covered w/ relevant materials 19 50 Some relevant materials 12 32 Little to no materials 7 18 Missing value 4 Total 42 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 28 Table 3. Classroom libraries # % Have 11 28 Do not 28 72 Missing 3 Total 42 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 29 Appendix A-6. Classroom Observations (data related to Mathematics subcomponent) Table 1. Teacher guidebook* # % Has a guide and consults 16 46 Has a guide and does not consult 8 23 No teacher guide 11 31 Missing 7 Total 42 *Applies to Literacy and Mathematics Table 2. Teacher lesson plan* # % Has plan, refers heavily 6 17 Has plan, refers occasionally 8 23 Has a plan, but does not use 12 34 No plan 9 26 Missing 7 Total 42 *Applies to Literacy and Mathematics Table 3. Mathematics textbooks for students # % * All students (100 percent) 14 67 More than half (50-99 %) 1 5 Less than half (1-49%) 0 0 No texts (0 %) 6 29 Missing 0 Total 21 *Percentages do not sum to 100 due to rounding AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 30 Figure 1. Teaching-learning strategies observed, n= 21 (classroom observations- MAT) 52% 58% 35% 25% 15% 8% 8% 35% 70% 5% 25% 8% 23%20% 3% 33% 3% 5% 8% 63% 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% Q13A1 Q13A2 Q13A3 Q13A4 Q13A5 Q13B1 Q13B2 Q13B3 Q13B4 Q13C Q13D Q13E Q13F Q13G1 Q13G2 Q13H Q13I Q13K Q13L Q13M Q13N Q13A1= WRITES ON BOARD FOR DISCUSSION Q13A2= WRITES ON BOARD TO ANSWER QUESTIONS Q13A3= WRITES ON BOARD TO HAVE STUDENTS COPY Q13A3= WRITES ON BOARD TO SUMMARIZE Q13A4= WRITES ON BOARD TO HAVE STUDENTS READ Q13B1= TEACHER DICTATES Q13B2=TEACHER READS ALOUD Q13B3= TEACHER ASKS STUDENTS QS INDV Q13B4= TEACHER ASKS STUDENTS QS GROUP Q13C= TEACHER DEMONSTRTES PHYSICALLY Q13D= TEACHER DEMONSTRATES WITH MATERIALS Q13E= TEACHER DRAWS Q13F= TEACHER USES POSTER Q13G= STUDENTS READ ALOUD Q13H= STUDENTS READ SILENTLY Q13I= STUDENTS ROLE PLAY Q13K= STUDENTS DEMONSTRATE PHYSICALLY Q13L= STUDENTS DEMONSTRATE WITH MATERIALS Q13M= STUDENTS DO SEATWORK Q13N= OTHER Table 4. Mathematics boxes # % Have 5 14 Do not have 32 86 Missing 5 Total 42 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Basic Education Program Annex A: ESP 31 Appendix A-7. Bibliography AMEX International (2012). USAID/ DR Evaluation Midterm Performance Evaluation, ppt presentation. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2010). Reporting Period: January 1, 2010- September 30, 2010. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2011). Reporting Period: January 1, 2011- May 31, 2011. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2012a). Reporting Period: January 1, 2012- May 31, 2012. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report (2012b). Report October 1, 2010- December 31, 2011. Effective Schools Program (USAID-PUCMM) Safe Schools Program- Summary of Results. (2010). Reporting Period: October 1, 2009- June, 2010. Effective Schools Program (USAID-PUCMM) Safe Schools Program- Annual Report. (2010). Reporting Period: October 1, 2009- September 30, 2010. MINERD (2011). Politicas y Estrategias para la Ensenanza y el Aprendizaje de la Lectura y Escritura de Matematica desde los Primeros Grados de la Educacion Basica- Propuesta de Intervencion. Organización de Estados Iberoamericanos (2010). Republica Dominica hacia el 2021, Documento Del Pais. PEF (2010). Reporte de Resultados Primero y Segundo Grado, PEF Matemática 2010. PEF (2010). Reporte de Resultados Primero y Segundo Grado, PEF Lengua 2010. PEF (2011). Frecuencia de la Cobertura de los Contenidos de libro de Texto- tercero y cuarto [project generated materials figures] PEF (2012). Informe Anual Componente de Gestión y Participación (Julio 2011- Julio 2012). TEF (2009). Reporte resumen semestral Octubre- Diciembre 2008, Enero- Marzo 2009. TEF (2008). Reporte trimestral: Julio- Septiembre, 2008. USAID (2009). Attachment B Program Description for Effective Schools Program. Dominican Republic. USAID (2010). Performance Monitoring & Evaluation- TIPS Constructing an Evaluation Report, No. 17, Washington, D.C. USAID (2011). USAID Education Strategy- Education Opportunity Through Learning, Washington, D.C. Valverde, G. & Naslund-Hadley, E. (2010). The State of Numeracy Education in Latin America and the Caribbean, Inter-American Development Bank, Washington D.C. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 1 Annex B: USAID/DR The Effective Schools Program-Analysis of the Program Database/Análisis de las Bases de Datos del Programa Submitted to: USAID/Dominican Republic Prepared by: Luis Domínguez, Ph.D. Education Data Specialist AMEX International & DevTech Systems, Inc. -DQXDU\ DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 2 Contenido Siglas usadas (Acrónimos) ..................................................................................................................................................... 5 1. Introducción......................................................................................................................................................................... 6 2. Distribución de la muestra de escuelas participantes en PEF............................................................................................ 7 2.1 Muestra general............................................................................................................................................................ 8 2.2 Distribución por sexo .................................................................................................................................................... 9 2.3 Muestra por Zona ....................................................................................................................................................... 12 2.4 Sobre-edad ................................................................................................................................................................. 13 2.5 Feminización según año de evaluación y grado ........................................................................................................ 14 2.6 Feminización y sobre-edad......................................................................................................................................... 15 2.6.1 Feminización según edad y grado ...................................................................................................................... 15 2.7 Muestra de Comparación ........................................................................................................................................... 16 2.8 Total casos por año, grado y área.............................................................................................................................. 16 2.9 Por sexo...................................................................................................................................................................... 17 2.10 Sobre-edad ............................................................................................................................................................... 18 3. Resultados en las Evaluaciones ....................................................................................................................................... 18 3.1 Resultados en Comprensión Lectora (CL) ................................................................................................................. 18 3.1.1 Resultados Generales (Año y Grado) ................................................................................................................. 18 3.1.2 Resultados en CL, según género........................................................................................................................ 20 3.1.3 Sobre-edad y CL ................................................................................................................................................. 21 3.1.4 Estrato (Zona Urbana o Rural), en Comprensión Lectora .................................................................................. 22 3.1.5 Distribución por rangos en CL............................................................................................................................. 22 3.1.6 Un análisis de regresión simple, para CL ........................................................................................................... 24 3.2 Resultados en Matemática (MT)................................................................................................................................. 25 3.2.1 Resultados Generales (Año y Grado) ................................................................................................................. 25 3.2.2 Resultados en MT, según género ....................................................................................................................... 26 3.3 Sobre-edad y MT ........................................................................................................................................................ 27 3.3.1 Estrato (Zona Urbana o Rural), en Matemática .................................................................................................. 27 3.3.2 Distribución por rangos en MT ............................................................................................................................ 28 3.3.3 Un análisis de regresión simple, para MT........................................................................................................... 31 4. Escuelas del Programa Versus Muestra de Comparación ............................................................................................... 32 Diferencias entre años, entre PEF y MC...................................................................................................................... 33 4.1 Sexo y resultados en PEF y MC................................................................................................................................. 34 4.2 Sobre-edad y MC........................................................................................................................................................ 34 APÉNDICE B-1 ..................................................................................................................................................................... 36 Tablas del Análisis de Regresión Lineal de Comprensión Lectora .................................................................................. 36 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 3 Regression Comprensión Lectora .................................................................................................................................... 36 Charts ........................................................................................................................................................................... 39 Regression Matemática .................................................................................................................................................... 41 Charts ........................................................................................................................................................................... 44 APÉNDICE B-2: Bibliographia .............................................................................................................................................. 47 Índice de Tablas Table 1: Muestra de sujetos en escuelas participantes, según año, grado y área académica ........................... 8 Table 2: Distribución sexo en CL....................................................................................................................... 9 Table 3: Muestra en Matemática, por sexo......................................................................................................10 Table 4: Muestra de sujetos en escuelas participantes, según año, grado y área académica ......................... 11 Table 5: Distribución de las proporción de casos por zona (rural/urbana) donde se ubica la escuela.............. 12 Table 6: Proporción de casos en sobre-edad, según año, grado y Área.......................................................... 13 Table 7: Índice de feminización por grado y año de la evaluación ................................................................... 14 Table 8: Índice de feminización, según edad y grado ......................................................................................15 Table 9: Cantidad de casos evaluados en la Muestra e Comparación, por área evaluada, grado y año de la evaluación .......................................................................................................................................................16 Table 10: Distribución por sexo en la muestra de comparación, según grado, área y año de evaluación ........ 17 Table 11: Índice de feminización, por grado, área y año.................................................................................. 17 Table 12: Proporción de sujetos en sobre-edad en la muestra de comparación, por curso, área evaluada y año ........................................................................................................................................................................18 Table 13: Promedio en Comprensión Lectora, por año y grado....................................................................... 18 Table 14: Promedios en CL, por sexo, según grado y año de evaluación........................................................ 20 Table 15: Relación con la sobre-edad, en CL, según grado y año................................................................... 21 Table 16: Estrato, según grado y año, en CL...................................................................................................22 Table 17: Proporción de estudiantes en los diferentes rangos, según año de evaluación, en CL .................... 22 Table 18: rangos de puntuaciones por año, para 4º grados, Comprensión Lectora ......................................... 23 Table 19: Resumen del modelo de regresión para Comprensión Lectora (Model Summaryb ) ......................... 24 Table 20: Coeficientes del análisis de regresión lineal en Comprensión Lectora (Coefficientsa ) ...................... 25 Table 21: Promedio en Matemática, por año y grado ......................................................................................26 Table 22: Diferencias de Promedios en MT, entre niñas y niños, según grado y año de evaluación ............... 26 Table 23: Relación con la sobre-edad, en MT, según grado y año .................................................................. 27 Table 24: Estrato, según grado y año, en MT..................................................................................................28 Table 25: Proporción de estudiantes en los diferentes rangos, según año de evaluación................................ 28 Table 26: rangos de puntuaciones por año, para 3º y 4º grados, Matemática.................................................. 30 Table 27: Sumario de los modelos detectados para MT (Model Summaryb ) .................................................... 31 Table 28: Coeficientes del análisis de regresión lineal en Matemática (Coefficientsa ) ...................................... 32 Table 29: Proporción de sujetos en sobre-edad en la muestra control, por curso, área evaluada y año.......... 33 Table 30: Diferencias en los resultados entre años en PEF y MC, según área y grado (3º y 4º)...................... 33 Table 31: Resultados comparativos por sexo, con relación a la MC, por Área y año....................................... 34 Table 32: Promedio en las pruebas, en casos con y sin sobre-edad, según tipo muestra (PEF o Comparasión), área, grado y año ............................................................................................................................................35 Índice de figuras Figure 1: Sample in CL, by grade and year........................................................................................................ 8 Figure 2: Sample in MT, by grade and year....................................................................................................... 9 Figure 3: Total de casos por grado y año, según sexo, en Comprensión Lectora............................................ 10 Figure 4: Muestra en MT, por grado y año, según sexo................................................................................... 10 Figure 5: Proporciones de casos según sexo, grado y año, en Comprensión Lectora ..................................... 11 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 4 Figure 6: Proporciones de casos según sexo, grado y año, en Comprensión Lectora ..................................... 11 Figure 7: Distribución de la muestra según zona donde se ubica la escuela, en CL ........................................ 12 Figure 8: Distribución de la muestra según zona donde se ubica la escuela, en MT ....................................... 12 Figure 9: Sobre-edad en CL, según año, por grado.........................................................................................14 Figure 10: Sobre-edad en MT, según año, por grado. .....................................................................................14 Figure 11: Feminización de la muestra, según grado y año de la evaluación .................................................. 15 Figure 12: Distribución de la muestra de comparación según grado, área evaluada y año de la evaluación ... 16 Figure 13: Feminización en la Muestra de Comparación, por año, según grado y área................................... 17 Figure 14: Promedios en CL, según grado y año de aplicación ....................................................................... 19 Figure 15: Distribución de la mediana de las puntuaciones y de los cuartiles intermedios en CL, según grado y año. ................................................................................................................................................................. 20 Figure 16: Medias en CL por sexo, grado y año ..............................................................................................21 Figure 17: Relación con la sobre-edad, en CL, según grado y año.................................................................. 21 Figure 18: Medias en CL en los estratos, según año y grado .......................................................................... 22 Figure 19: Distribución de rangos de puntuaciones en CL, según año (todos los grados) ............................... 23 Figure 20: Distribución de rangos de puntuaciones en CL para 4º grado, según año ...................................... 24 Figure 21: Promedios en MT, según grado y año de aplicación....................................................................... 26 Figure 22: Medias en MT por sexo, grado y año..............................................................................................27 Figure 23: Relación con la sobre-edad, en MT, según grado y año ................................................................. 27 Figure 24: Medias en MT en los estratos, según año y grado.......................................................................... 28 Figure 25: Distribución por año de los rangos que ocupan los estudiantes en MT........................................... 29 Figure 26: Distribución de rangos de puntuaciones en MT, por año ................................................................ 29 Figure 27: Porcentajes de casos en los diferentes rangos de puntuación, en 3º grado, según año; para Matemática......................................................................................................................................................30 Figure 28: Porcentajes de casos en los diferentes rangos de puntuación, en 3º grado, según año; para Matemática......................................................................................................................................................31 Figure 29: Escuelas PEF vs MC, por Área, grado y año.................................................................................. 33 Figure 30: Resultados por sexo en PEF vs MC, según Área y año,................................................................. 34 Figure 31: Promedios comparados con MC, por sobre-edad, área, grado y año ............................................. 35 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 5 Siglas usadas (Acrónimos) CL Comprensión Lectora MC Muestra de Comparación MT Matemática PEF Programa Escuelas Efectivas PUCMM Pontificia Universidad Católica Madre y Maestra USAID United States Agency for International Development AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 6 Resultados 1. Introducción El presente informe presenta el análisis de información derivado de las bases de datos suministradas por el Programa Escuelas Efectivas (PEF), ejecutado por la Pontificia Universidad Católica Madre y Maestra (PUCMM) y ejecutados con fondos de USAID, bajo el Cooperative Agreement No. 517-A-00-10-00102-00. Los datos consisten en las medias de las puntuaciones (proporción de respuestas correctas) alcanzadas por estudiantes de escuelas del PEF, tanto en Comprensión Lectora (CL) como en Matemática (MT), así como datos generales de identificación de estudiantes y escuelas (edad, sexo, localización geográfica, año de la evaluación), para los grados 1º a 4º de la Educación Básica, y en los años 2008 al 2011. También se analizó una muestra de comparación de escuelas que no estaban en el PEF, tomada solamente para 3º y 4º grados y en los años 2010 y 2011. Los análisis comienzan con una revisión de la distribución de escuelas y estudiantes que participan en el PEF, analizados por grado y año en que se produce la evaluación, según las distintas variables de comparación: Sexo, zona (rural/urbana), edad, sobre-edad e índice de feminización. Más adelante, se realiza un análisis más profundo sobre los resultados de las evaluaciones cognoscitivas en las áreas objeto del Programa: Los datos se analizan, para cada área, por grado y año de evaluación, agregados por sexo, sobre￾edad, zona o “estrato” y rangos de puntuaciones. Se hacen estudios de regresión linear simple en cada caso. Por último, se hace un análisis comparativo de los resultados en CL y MT de los años 2010 y 2011 en 3º y 4º grados, con la muestra de comparación (MC) de escuelas no-PEF, por sexo y sobre-edad. Los principales resultados son los siguientes: a) Se han evaluado 75,354 estudiantes del PEF, en Comprensión Lectora y Matemática, en los grados 1º a 4º, de los años 2008 al 2011: No todos los años están todos los grados ni ambas áreas. También se evaluaron 9,418 niños y niñas (NN) de escuelas que no estaban en el PEF, en lo que se denominó “Muestra de Comparación”(MC), en los grados 3º y 4º, y los años 2010 y 2011, tanto para CL como para MT. b) El porcentaje de niñas (femenino) en aulas tiende a incrementarse al aumentar los años de escolaridad. c) La sobre-edad (dos años por encima de la edad normativa), tiende a aumentar con el grado académico; y es un fenómeno que interesa prioritariamente más a los varones (niños) d) El programa PEF, al avanzar en los años de ejecución, tiende a hacerse más “urbano”: Integra más escuelas clasificadas como del sector urbano que el rural. e) Se percibe un incremento general en el rendimiento en Comprensión Lectora año tras año, en el 4º grado y en el 1º grado. f) Por lo general, las niñas tienen mejor resultado en CL que los niños, la mayor parte de ellos estadísticamente significativos. g) En los grados 3º y 4º, NN en sobre-edad tienen significativamente resultados más bajos que los que no están en sobre-edad, en CL. h) Para 2º, 3º y 4º grados, NN del sector urbano tienen mejor resultado que los del sector rural, en los dos últimos años (2010 y 2011), en CL. i) Para CL, se nota un significativo incremento de NN que ocupan los tercios medio y superior de la distribución de las calificaciones sobre 100 puntos, y una diminución significativa de los que están en el tercio inferior, desde el 2008 al 2011. j) El análisis de regresión de los resultados en CL muestra que las variables año del programa, sexo, sobre-edad, zona rural/urbana y grado académico son significativas para predecir el rendimiento de NN en las evaluaciones. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 7 El perfil de mayor éxito es el de una niña de cuarto grado matutino de una escuela urbana que está en la edad normativa para su grado, durante el año 2011. k) Para MT se observa que el progreso en los resultados de las evaluaciones parece haber alcanzado una cima en los años 2009 y 2010, y que para el 2011 muestra haberse detenido o hasta retrocedido. l) Al igual que en CL, las niñas tiene significativamente mejores resultados que los niños en MT. m) Lo mismo se confirma con respecto a la sobre-edad en 3º y 4º grados: NN en sobre-edad tienen resultados más bajos en esos grados que los sin sobre-edad. n) El diferencia en el rendimiento por zona urbana/rural en MT es semejante al de CL, donde la zona urbana presenta mejor rendimiento; aunque a ciertos niveles (3º en el 2008, 1º en el 2010 y 2º en el 2011), la situación era inversa a la observada más tarde: Los resultados de la zona rural eran superiores a los de zona urbana. o) La distribución por rangos de puntuaciones en MT, a diferencia de CL, muestra que los años 2009 y 2010 los resultados fueron significativamente más altos (más estudiantes pasan de un rango al superior) que en el 2008 y el 2011. Esto es bastante lineal en 3º grado, pero en 4º grado lo que se observa es que, luego del 2008, la proporción de NN en el rango superior (más del 66% de aciertos en la evaluación) se mantiene inalterada (9%). p) El análisis de regresión arroja resultados semejantes a los de CL, aunque con un menor nivel de significatividad, sobre todo en la variable “Tanda”. El perfil es semejante al de CL. Se sospecha que algunas variables (año de la evaluación, por ejemplo) no son lineales, como se observó en los resultados disgregados. q) La comparación con la “Muestra de Comparación”, para 3º y 4º en los años 2010 y 2011, muestra los siguientes resultados: a. En todos los casos, los resultados del PEF son estadísticamente superiores a los de la MC. b. El análisis de los cambios entre un año y otro no muestra grandes diferencias en el comportamiento de la MC versus PEF: En algunos casos (En CL para 3º grado MC y 4º grado PEF) el cambio es significativo, pero en otros no se observa significatividad, y aún más, en MT en 3º grado para NN en el PEF, los resultados son SUPERIORES en el 2010 que en el 2011. c. Tanto en el PEF como en la MC, las niñas tienen mejores resultados en CL que los niños; y en MT las niñas del PEF aventajan a los niños, mientras que en la MC no hay diferencia entre los sexos en el resultado en MT. d. La sobre-edad está asociada a rendimiento más bajo en el PEF, y que en la MC se observa lo mismo, excepto en CL en 3º y 4º grados en el 2010, y en MT en 3º grado del 2011, donde no hay diferencias significativas debidas a la sobre-edad. 2. Distribución de la muestra de escuelas participantes en PEF Los siguientes datos muestran la distribución de la muestra bajo diferentes criterios (general, por sexo, por sobre￾edad…). La mayor parte de las tablas presentan la proporción (porcentaje) de casos en el criterio considerado; y la significatividad estadística de la diferencia entre esas proporciones. En las tabla, las casillas sombreadas muestran la significatividad de las proporciones de casos al compararse entre sujetos en cada uno de los criterios: Cuando la proporción de sujetos del criterio a la izquierda es mayor que el del criterio a la derecha, se marca con un color naranja, indicando que en ese grado y año el criterio a la derecha es significativamente inferior que el de la izquierda. Lo contrario se indica con un sombreado azul sobre el criterio significativamente más elevado. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 8 2.1 Muestra general Table 1: Muestra de sujetos en escuelas participantes, según año, grado y área académica 2008 2009 2010 2011 Total Comprensión Lectora 1ro 191 269 460 2do 379 291 670 3ro 5,454 6,039 11,493 4to 1,832 1,444 5,187 5,797 14,260 Total 1,832 1,444 11,211 12,396 26,883 Matemática 1ro 393 544 937 2do 355 358 713 3ro 3,839 5,189 7,262 8,345 24,635 4to 2,518 4,537 7,583 7,548 22,186 Total 6,357 9,726 15,593 16,795 48,471 TOTAL 1ro 584 813 1,397 2do 734 649 1,383 3ro 3,839 5,189 12,716 14,384 36,128 4to 4,350 5,981 12,770 13,345 36,446 Total 8,189 11,170 26,804 29,191 75,354 En total se recibió información de 75,354 estudiantes, de los cuales el 36% son del componente de Comprensión Lectora, mientras el resto (48,471) son de Matemática. Para los años 2008 y 2009 no hubo muestra en 1º y 2º grados; como tampoco para Comprensión Lectora (CL) en 3º grado de esos años. Figure 1: Sample in CL, by grade and year AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 9 Figure 2: Sample in MT, by grade and year 2.2 Distribución por sexo Table 2: Distribución sexo en CL 2008 2009 2010 2011 Total Grado 1ro Niñas 106 134 240 Niños 84 135 219 2do Niñas 179 131 310 Niños 200 160 360 3ro Niñas 2562 2790 5352 Niños 2883 3206 6089 4to Niñas 972 745 2480 2766 6963 Niños 818 674 2698 3020 7210 Total Niñas 972 745 5327 5821 12865 Niños 818 674 5865 6521 13878 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 10 Figure 3: Total de casos por grado y año, según sexo, en Comprensión Lectora Table 3: Muestra en Matemática, por sexo 2008 2009 2010 2011 Total Grado 1ro Niñas 226 307 533 Niños 167 236 403 2do Niñas 196 203 399 Niños 157 155 312 3ro Niñas 1922 2463 3420 3952 11757 Niños 1765 2474 3820 4339 12398 4to Niñas 1297 2283 3790 3718 11088 Niños 1145 2153 3782 3811 10891 Total Niñas 3219 4746 7632 8180 23777 Niños 2910 4627 7926 8541 24004 Figure 4: Muestra en MT, por grado y año, según sexo Al analizar la muestra en función del porcentaje de niñas/niños, se detectan tendencias interesantes: Hay más niñas en 1º grado que niños, en los grupos de CL evaluados en el año 2010. En MT se observa que en casi todos los cursos, excepto en 3º, la proporción de niñas es significativamente mayor que la de niños, en el 2010, mientras que en el 3º es AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 11 mayor el porcentaje de varones. En el 2011 se aprecia el mismo fenómeno, con la diferencia que para 4º no hay diferencias significativas, aunque numéricamente el porcentaje de niñas es menor que el de varones. Table 4: Muestra de sujetos en escuelas participantes, según año, grado y área académica 2008 2009 2010 2011 Total Niñas Niños Niñas Niños Niñas Niños Niñas Niños Niñas Niños Comprensión Lectora 1ro 56% 44% 50% 50% 52% 48% 2do 47% 53% 45% 55% 46% 54% 3ro 47% 53% 47% 53% 47% 53% 4to 54% 46% 53% 47% 48% 52% 48% 52% 49% 51% Matemática 1ro 58% 42% 57% 43% 57% 43% 2do 55% 45% 57% 43% 56% 44% 3ro 52% 48% 50% 50% 47% 53% 48% 52% 49% 51% 4to 53% 47% 51% 49% 50% 50% 49% 51% 50% 50% Total 1ro 57% 43% 54% 46% 55% 45% 2do 51% 49% 51% 49% 51% 49% 3ro 52% 48% 50% 50% 47% 53% 47% 53% 48% 52% 4to 54% 46% 52% 48% 49% 51% 49% 51% 50% 50% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 5: Proporciones de casos según sexo, grado y año, en Comprensión Lectora Figure 6: Proporciones de casos según sexo, grado y año, en Comprensión Lectora AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 12 2.3 Muestra por Zona Table 5: Distribución de las proporción de casos por zona (rural/urbana) donde se ubica la escuela 2008 2009 2010 2011 Rural Urbano Rural Urbano Rural Urbano Rural Urbano Comprensión Lectora 1ro 43% 57% 41% 59% 2do 50% 50% 44% 56% 3ro 38% 62% 37% 63% 4to 47% 53% 59% 41% 37% 63% 35% 65% Matemática 1ro 42% 58% 34% 66% 2do 25% 75% 16% 84% 3ro 41% 59% 51% 49% 34% 66% 35% 65% 4to 44% 56% 54% 46% 33% 67% 33% 67% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 7: Distribución de la muestra según zona donde se ubica la escuela, en CL Figure 8: Distribución de la muestra según zona donde se ubica la escuela, en MT En el transcurso de los años, la muestra tiende a hacerse más “urbana”, aunque en 3º en Matemática y en 2º en CL la tendencia es a la inversa: tiende a ser más “rural”. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 13 2.4 Sobre-edad Los niños evaluados se clasificaron como “en sobre-edad”, cuando su edad en años cumplidos era de 2 años o más por encima de la edad normativa para el grado (6 años para primero). La siguiente tabla muestra la distribución de sobre￾edad en la muestra, por año, grado y área académica evaluada. Se ha enfatizado el efecto “año de la evaluación” sobre el de grado, para intentar aproximarse a un “efecto programa”. Table 6: Proporción de casos en sobre-edad, según año, grado y Área 1ro 2do 3ro 4to No Sí No Sí No Sí No Sí Comprensión Lectora 2008 84% 16% 2009 89% 11% 2010 97% 3% 94% 6% 88% 12% 87% 13% 2011 98% 2% 94% 6% 89% 11% 88% 12% Matemática 2008 85% 15% 82% 18% 2009 86% 14% 84% 16% 2010 100% 0% 91% 9% 89% 11% 86% 14% 2011 98% 2% 97% 3% 89% 11% 87% 13% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, La sobre-edad es más marcada en 3º y 4º grados para matemática, en los años 2008 y 2009; y para el 2010 en 4º. En Comprensión Lectora esta se presenta solamente en el 2008 en 4º grado. Este dato es interesante, porque señala que la sobre-edad tiende a disminuir con los años de evaluación, sugiriendo que el programa podría estar interviniendo en ese sentido. La serie, para los grados 1º y 2º es corta (solamente dos años) para poder pronosticar tendencias. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 14 Figure 9: Sobre-edad en CL, según año, por grado Figure 10: Sobre-edad en MT, según año, por grado. 2.5 Feminización según año de evaluación y grado El siguiente cuadro muestra el índice de feminización en cada grado, según año de la evaluación. Este índice es la relación (ratio) de número de niñas dividido número de niños. Si la proporción de niñas es mayor que la de niños, el índice será superior a 1.00. Cuando el índice es igual o mayor de 1.20 se ha indicado con color verde (Ejemplo: 1.32), indicando una significativa proporción de mujeres mayor que de hombres. Cuando la proporción de niños supera a la de niñas (0.80 ó menos), se indica con caracteres rojos (ejemplo: 0.45). En la siguiente tabla se muestra el índice de feminización para todas las edades, según grado y año en que se realizó la evaluación. Solamente se destaca una mayor feminización en 1º grado, en el 2010. Para el resto de los años, el índice cae dentro del rango de lo normal. Table 7: Índice de feminización por grado y año de la evaluación Año de Evaluación 2008 2009 2010 2011 Grado 1ro 1.32 1.19 2do 1.05 1.06 3ro 1.09 1.00 0.89 0.89 4to 1.16 1.07 0.97 0.95 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 15 Figure 11: Feminización de la muestra, según grado y año de la evaluación 2.6 Feminización y sobre-edad 2.6.1 Feminización según edad y grado En el siguiente cuadro se reporta el índice de feminización, con respecto a la edad y el grado. La zona sombreada es la correspondiente a la edad normativa para el grado reportado (+/- 1 año para la edad normativa, que es de 6 años para primer grado). La zona por debajo de esta área sombreada es el área de sobre-edad para el grado considerado (2 años o más por encima de la edad normativa); mientras el área por encima de la zona sombreada es la de infra-edad (2 años o más POR DEBAJO de la edad normativa. Nótese que, a parte del 1º grado, la sobre-edad es un fenómeno masculino, mientras el estar en la edad normativa es más bien característico de las niñas. Table 8: Índice de feminización, según edad y grado Grado 1ro 2do 3ro 4to Edad (años cumplidos) 5 0.83 Área de infraedad 6 1.34 0.75 1.00 7 1.22 1.48 1.72 2.54 8 0.67 0.95 1.30 1.64 9 1.20 0.60 0.95 1.39 10 1.33 0.32 0.62 1.04 11 0.33 0.49 0.69 12 1.00 0.45 0.57 13 0.36 0.47 14 Área de sobre-edad 0.43 0.38 15 0.51 0.35 16 0.47 0.40 17 1.00 0.25 Algunas observaciones a este cuadro: a) Se eliminaron de la base de datos sujetos con edades superiores a los 17 años (unos 12 casos, del total de 75,000). AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 16 b) Del cuadro de análisis se eliminaron las casillas en que había solamente uno o dos casos en cualquiera de los sexos, ya que una variación de uno daba indicadores de feminización significativamente altos o bajos, distorsionando el indicador. 2.7 Muestra de Comparación Para los años 2010 y 2011 se trabajó una “Muestra de Comparación” (MC), consistente en casos de 37 escuelas del sector público. Las siguientes tablas resumen los datos descriptivos de esta muestra. 2.8 Total casos por año, grado y área Table 9: Cantidad de casos evaluados en la Muestra e Comparación, por área evaluada, grado y año de la evaluación 2010 2011 Total Comprensión Lectora 3ro 1188 1141 2329 4to 1248 1221 2469 Total 2436 2362 4798 Matemática 3ro 1278 1185 2463 4to 1284 873 2157 Total 2562 2058 4620 Figure 12: Distribución de la muestra de comparación según grado, área evaluada y año de la evaluación AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 17 2.9 Por sexo Table 10: Distribución por sexo en la muestra de comparación, según grado, área y año de evaluación Año de Evaluación 2010 2011 Total Niñas Niños Niñas Niños Niñas Niños Comprensión Lectora 3ro 47.3% 52.7% 49.7% 50.3% 48.5% 51.5% 4to 49.1% 50.9% 50.2% 49.8% 49.7% 50.3% Matemática 3ro 49.6% 50.4% 46.2% 53.8% 48.0% 52.0% 4to 48.9% 51.1% 50.9% 49.1% 49.7% 50.3% Total 3ro 48.5% 51.5% 47.9% 52.1% 48.2% 51.8% 4to 49.0% 51.0% 50.5% 49.5% 49.7% 50.3% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, La distribución por sexo es relativamente homogénea, según se muestra en la tabla de índices de feminización por grado y año. La distribución por género es bastante semejante a la del PEF, con una ligera tendencia a incrementase el número de niñas en 4º grado, en el 2011 respecto al 2010, pero lo contrario sucede en 3º grado, siempre en MT. Aunque no sea estadísticamente significativo, la tendencia observada es hacia una mayor feminización, en el transcurso de los años. Table 11: Índice de feminización, por grado, área y año 2010 2011 Comprensión Lectora 3ro 0.90 0.99 4to 0.96 1.01 Matemática 3ro 0.98 0.86 4to 0.96 1.04 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 13: Feminización en la Muestra de Comparación, por año, según grado y área AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 18 2.10 Sobre-edad Table 12: Proporción de sujetos en sobre-edad en la muestra de comparación, por curso, área evaluada y año 3ro 4to No Sí No Sí Comprensión Lectora 2010 86% 14% 100% 0% 2011 87% 13% 85% 15% Matemática 2010 86% 14% 88% 12% 2011 86% 14% 84% 16% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, La sobre-edad en la MC oscila entre 12% y 16% (media para MT = 13.7% y 10.03 para CL). En el 2010, en 4º grado, en CL, no se detectó ningún caso de sobre-edad. 3. Resultados en las Evaluaciones 3.1 Resultados en Comprensión Lectora (CL) 3.1.1 Resultados Generales (Año y Grado) El promedio general en CL es de 47% de respuestas correctas. Por lo general, los porcentajes más bajos son en 4º grado, donde los mismos no superan el 44% en ningún año. En la tabla siguiente, se observa que para 4º grado, las puntuaciones obtenidas en promedio cada año son significativamente mejores que en el año precedente. Lo mismo se observa para 1º Grado entre el 2010 y el 2011 (las zonas sombreadas significan que las puntuaciones son significativamente mayores que las del período precedente.). No se observan variaciones significativas entre el 2010 y el 2011 para 2º y 3º grados. Table 13: Promedio en Comprensión Lectora, por año y grado Año de Evaluación 2008 2009 2010 2011 Total 1ro .55 .61 .58 2do .57 .55 .56 3ro .52 .52 .52 4to .37 .39 .43 .44 .42 Total .37 .39 .48 .49 .47 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 19 Figure 14: Promedios en CL, según grado y año de aplicación Otro dato importante lo muestra el siguiente gráfico, donde se analiza la dispersión de las puntuaciones y la distribución de la mediana con los cuartiles Q2 y Q3. Se nota la amplitud de la distribución (que siempre va de 0% al 100%), y la fuerte presencia de valores extremos (outliers), sobre todo en 3º y 4º grados, lo que indica una gran variabilidad en los casos evaluados, reforzando la idea de que no son poblaciones homogéneas. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 20 Figure 15: Distribución de la mediana de las puntuaciones y de los cuartiles intermedios en CL, según grado y año. 3.1.2 Resultados en CL, según género La comparación de las medias de niñas y niños muestra que las niñas siempre tienen promedios más altos que los varones. Estas diferencias son significativas, excepto para 1º grado, aunque aún aquí las niñas tienen resultados numéricos superiores a los varones. Table 14: Promedios en CL, por sexo, según grado y año de evaluación Año de Evaluación 2008 2009 2010 2011 Niñas Niños Niñas Niños Niñas Niños Niñas Niños 1ro .56 .54 .61 .60 2do .59 .55 .58 .53 3ro .55 .49 .55 .49 4to .40 .34 .43 .36 .46 .40 .48 .41 Total .40 .34 .43 .36 .51 .45 .52 .46 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 21 Figure 16: Medias en CL por sexo, grado y año 3.1.3 Sobre-edad y CL Los resultados considerando la variable sobre-edad, muestran que en 3º y 4º grados, las puntuaciones medias en CL son significativamente mayores en los sujetos SIN sobre-edad, mientras que en 1º y 2º grados este efecto no es tan marcado, aunque numéricamente son cifras más elevadas. Table 15: Relación con la sobre-edad, en CL, según grado y año Año de Evaluación 2008 2009 2010 2011 Sobre-edad Sobre-edad Sobre-edad Sobre-edad No Sí No Sí No Sí No Sí 1ro 0.55 0.59 0.61 0.56 2do 0.57 0.61 0.56 0.49 3ro 0.52 0.48 0.53 0.46 4to 0.38 0.30 0.41 0.30 0.43 0.37 0.45 0.39 Total .38 .30 .41 .30 .48 .43 .49 .42 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda. Figure 17: Relación con la sobre-edad, en CL, según grado y año AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 22 3.1.4 Estrato (Zona Urbana o Rural), en Comprensión Lectora El estrato donde se ubica la escuela tiene relevancia en las evaluaciones de los años 2010 y 2011, donde los estudiantes de escuelas clasificadas como “urbanas” obtienen resultados significativamente superiores a los de los de zona rural, excepto en 1º grado. Table 16: Estrato, según grado y año, en CL Año de Evaluación 2008 2009 2010 2011 Rural Urbano Rural Urbano Rural Urbano Rural Urbano 1ro 0.55 0.55 0.58 0.62 2do 0.55 0.59 0.52 0.58 3ro 0.50 0.53 0.49 0.53 4to 0.38 0.36 0.39 0.40 0.40 0.44 0.43 0.45 Total .38 .36 .39 .40 .46 .49 .47 .50 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda Figure 18: Medias en CL en los estratos, según año y grado 3.1.5 Distribución por rangos en CL Observando la tabla de dispersión de las puntuaciones, llama la atención la gran varianza en las mismas, donde el rango de las calificaciones va de 0% a 100%, aunque la media se concentra en puntuaciones entre 30% a 60%. Para hacer un análisis de cómo se mueven estos rangos a lo largo de los años, se dividió la muestra en tres grupos: Grupo 1: Con puntuaciones de 0 a 33%, Grupo 2: con puntuaciones superiores a 33% hasta 66%; y Grupo 3 con puntuaciones superiores a 66%. Table 17: Proporción de estudiantes en los diferentes rangos, según año de evaluación, en CL Año de Evaluación 2008 2009 2010 2011 Total Menos del 33% 42.9% 36.6% 21.8% 20.5% 23.4% Entre 33% y 66% 49.8% 53.4% 60.3% 59.8% 59.0% Más del 66% 7.3% 10.0% 18.0% 19.7% 17.6% Total 100.0% 100.0% 100.0% 100.0% 100.0% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Verde: Significativamente mayor que los anteriores, excepto el que le queda directamente a la izquierda. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 23 La siguiente tabla muestra la significancia de la diferencia entre las proporciones observadas; y su interpretación es que para el primer grupo (menos del 33%), la proporción de casos en el 2008 es significativamente mayor que en los siguientes años; e igual sucede en el 2009; mientras no es significativa la diferencia entre el 2010 y el 2011. Para los casos del grupo superior, cada año es significativamente mayor la proporción de sujetos en ese grupo, con respecto a todos los años anteriores. Por último, el grupo “medio” (33 – 66) presenta dos grandes supergrupos: 2008 + 2009 y 2010 + 2010, donde no hay diferencia al interior de ellos (intergrupo: 2010 no se diferencia de 2011), pero sí entre ellos (2010+2011 tiene mayor proporción que en los dos años anteriores). Figure 19: Distribución de rangos de puntuaciones en CL, según año (todos los grados) Lo observado confirma que a medida que avanzan los años en la evaluación, el rango de más de 66% y el del grupo medio (entre 33 y 66%) se incrementa, reduciéndose el de “Menos de 33%”. Dado que para los grados 1º a 3º solamente se dispone de información de dos años, se realizó el análisis de la distribución por rangos para el 4º grado, solamente, que sí presenta la serie 2008 al 2011 completa. Table 18: rangos de puntuaciones por año, para 4º grados, Comprensión Lectora Año de Evaluación 2008 2009 2010 2011 4to <= 0.33 43% 37% 33% 30% 0.34 a 0.66 50% 53% 55% 55% 0.67 y más 7% 10% 12% 15% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Verde: Significativamente mayor que los anteriores, excepto el que le queda directamente a la izquierda. Morado: Significativamente mayor que el primer valor a la izquierda, Amarillo: Significativamente mayor que el último valor a la derecha. Los resultados con esta serie indican que: a) El grupo de más bajo rendimiento es significativamente mayor en el 2008 que en todos los siguientes años. Para el 2011, el grupo de más bajo rendimiento es significativamente menor que en todos los anteriores años. b) El grupo de rendimiento medio (0.37 a 0.66), en el 2010 y el 2011 es mayor que en el 2008. c) El grupo de rendimiento más elevado (más de 0.66) en el 2011 es mayor que en todos los años anteriores, y el 2009 y 2010 son mayores que en el 2008. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 24 Figure 20: Distribución de rangos de puntuaciones en CL para 4º grado, según año En síntesis, se puede concluir que la proporción de estudiantes que alcanza rangos superiores a los obtenidos anteriormente en sus puntuaciones varía con los años de intervención en el programa, sobre todo los de rangos medios e inferiores. 3.1.6 Un análisis de regresión simple, para CL Se realizó un análisis de regresión lineal simple, con la introducción de Promedio en CL, como variable dependiente; y las variables Año de evaluación (2008 al 2011), Estrato (0=Rural, 1=Urbano), Sobre-edad (0=No, 1=Sí), Sexo (0 Femenino, 1 Masculino), Tanda (0=Matutina, 1=Vespertina) y Grado (años de escolaridad en Básica) como explicativas. En anexo se presenta todo el output de dicho análisis, y aquí se describen los principales resultados. a) El modelo explica cerca del 35% de la varianza observada. b) Todas las variables introducidas se correlacionan con la variable dependiente (porcentaje de respuestas correctas en la prueba de CL) c) El modelo en sí es altamente significativo, con p<0.0005 d) Todas las variables introducidas resultaron ser significativas para explicar el modelo. Table 19: Resumen del modelo de regresión para Comprensión Lectora (Model Summaryb ) Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .346a .120 .120 .17311 .120 590.892 6 25993 .000 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 25 Los coeficientes del modelo se presentan en la siguiente tabla: Table 20: Coeficientes del análisis de regresión lineal en Comprensión Lectora (Coefficientsa ) Model Unstandardized Coefficients Standardized Coefficients t Sig. 95.0% Confidence Interval for B B Std. Error Beta Lower Bound Upper Bound 1 (Constant) -41.992 2.647 -15.862 .000 -47.180 -36.803 Año de Evaluación .021 .001 .098 16.148 .000 .019 .024 Estrato (Dummy) .027 .002 .071 12.044 .000 .022 .031 Sobre-edad -.050 .003 -.087 -14.885 .000 -.057 -.044 Grado -.069 .002 -.239 -39.750 .000 -.073 -.066 Sexo -.054 .002 -.145 -24.755 .000 -.058 -.049 Tanda (Dummy) -.012 .002 -.032 -5.492 .000 -.016 -.008 a. Dependent Variable: Promedio Los coeficientes detallan la fórmula predictora de la puntuación real obtenida con base en las variables del modelo. Lo que estos coeficientes significan (tomando los coeficientes no-estandarizados) es que, a paridad de los demás factores, se encuentra que: a) Por cada año del Programa hay una ganancia de un 2% en la calificación final b) Por estar en la escuela urbana se recibe un 3%. c) No estar en sobre-edad, significa una ventaja de un 5%. d) Cada año de escolaridad otorga un 7% e) Por el hecho de ser niña, se gana 5% en la calificación final. f) Los estudiantes de la tanda matutina ganan un 1% La fórmula predictora se convierte en: Pp = -41.992 + (0.021 * Año de Evaluación) + (0.027 * Estrato) - (0.050 * Sobre￾edad) - (0.069 * Grado) - (0.054 * Sexo) - (0.012 * Tanda). Traducido en un lenguaje figurativo, el perfil de los más aventajados es el de una niña de cuarto grado matutino de una escuela urbana que está en la edad normativa para su grado, durante el año 2011. 3.2 Resultados en Matemática (MT) 3.2.1 Resultados Generales (Año y Grado) El promedio general en MT es de 40% de respuestas correctas. Por lo general, los porcentajes más bajos son en 3º grado, donde los mismos no superan el 37% en ningún año. En la tabla siguiente, se observa que para 3º grado, las puntuaciones obtenidas en los años 2009 y 2010 son significativamente mejores que en los años 2008 y el 2011. Para 4º grado, los años 2009 al 2010 presentan promedios significativamente más elevados que para el 2008, pero entre ellos no hay diferencias: Se mantienen estables en un 44% de logro en los 3 años. No se observan diferencias estadísticamente significativas entre el 2010 y el 2011 para 1º y 2º grados. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 26 Table 21: Promedio en Matemática, por año y grado Año de Evaluación 2008 2009 2010 2011 Total 1ro 0.56 0.58 0.57 2do 0.61 0.60 0.61 3ro 0.35 0.37 0.37 0.36 0.36 4to 0.41 0.44 0.44 0.44 0.44 Total 0.38 0.40 0.41 0.41 0.40 Leyenda: Verde: Significativamente mayor que el año 2008 Y el año 2011. Marrón claro: Significativamente mayor que la cifra que le queda a la izquierda Figure 21: Promedios en MT, según grado y año de aplicación 3.2.2 Resultados en MT, según género La comparación de las medias de niñas y niños muestra que las niñas siempre tienen promedios más altos que los varones. Estas diferencias son significativas, excepto para 1º grado en el 2010 y 2º grado en el 2011, aunque en ambos casos las niñas tienen resultados numéricos superiores a los varones, en un comportamiento semejante al observado en CL. Table 22: Diferencias de Promedios en MT, entre niñas y niños, según grado y año de evaluación Año de Evaluación 2008 2009 2010 2011 Niñas Niños Niñas Niños Niñas Niños Niñas Niños 1ro 0.57 0.54 0.61 0.54 2do 0.64 0.57 0.62 0.58 3ro 0.36 0.35 0.38 0.37 0.38 0.36 0.36 0.35 4to 0.42 0.41 0.45 0.43 0.45 0.43 0.45 0.43 Total 0.39 0.37 0.41 0.40 0.43 0.40 0.42 0.39 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 27 Figure 22: Medias en MT por sexo, grado y año 3.3 Sobre-edad y MT Los resultados, considerando la variable sobre-edad, muestran que en 4º grado, las puntuaciones medias en MT son significativamente mayores en los sujetos sin sobre-edad, y lo mismo se observa en 3º grado, excepto en el año 2009 , mientras que en 1º y 2º grados este efecto no es tan marcado. Table 23: Relación con la sobre-edad, en MT, según grado y año Año de Evaluación 2008 2009 2010 2011 Sobre-edad Sobre-edad Sobre-edad Sobre-edad No Sí No Sí No Sí No Sí 1ro 0.56 0.58 0.66 2do 0.61 0.60 0.60 0.52 3ro 0.36 0.33 0.37 0.36 0.37 0.35 0.36 0.33 4to 0.42 0.38 0.45 0.40 0.45 0.40 0.44 0.41 Total .38 .35 .41 .38 .42 .38 .41 .38 Figure 23: Relación con la sobre-edad, en MT, según grado y año 3.3.1 Estrato (Zona Urbana o Rural), en Matemática El estrato donde se ubica la escuela tiene relevancia en casi todas las evaluaciones, pero con resultados muy diferentes a los observados en CL. En 1º grado, alumnos de escuelas clasificadas como rurales en el 2010 obtienen promedios significativamente superiores a los de la zona urbana, pero la situación se revierte en el 2011. Para 2º grado, el comportamiento es diametralmente opuesto al del 1º grado: Urbano en el 2010 y rural en el 2012 son los que obtienen AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 28 promedios significativamente superiores. Para 3º grado se observan diferencias en el 2008 (Rural es superior), y en el 2011 (Urbano es superior). Por último, en 4º grado, todos los años los alumnos de zona urbana tienen resultados superiores a los de zona rural. Table 24: Estrato, según grado y año, en MT Año de Evaluación 2008 2009 2010 2011 Rural Urbano Rural Urbano Rural Urbano Rural Urbano 1ro 0.60 0.53 0.52 0.62 2do 0.57 0.62 0.69 0.59 3ro 0.37 0.35 0.37 0.37 0.37 0.37 0.34 0.36 4to 0.40 0.43 0.43 0.45 0.43 0.44 0.43 0.44 Total .38 .38 .40 .41 .41 .42 .39 .42 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 24: Medias en MT en los estratos, según año y grado 3.3.2 Distribución por rangos en MT Los siguientes reproducen el análisis por rangos de puntuaciones realizado para CL, esta vez para los resultados en Matemática. Table 25: Proporción de estudiantes en los diferentes rangos, según año de evaluación Año de Evaluación 2008 2009 2010 2011 Total Menos del 33% 38.5% 33.4% 32.2% 35.2% 34.3% Entre 33% y 66% 55.9% 59.2% 59.3% 56.9% 58.0% Más del 66% 5.6% 7.3% 8.5% 7.9% 7.7% Total 100.% 100.% 100.% 100.% 100.% Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Verde: Significativamente mayor que los años 2008 y 2011. Amarillo: Significativamente mayor que el 2008 La tabla de significatividad de las diferencias muestra que los casos de puntuaciones superiores al 66% son más frecuentes posterior al 2008, pero el 2012 solamente es significativamente diferente al 2008, mientras que en los otros años (2009 y 2010) estos son significativamente diferentes a los anteriores años. La proporción de casos con puntuaciones inferiores al 33% es significativamente mayor en el 2008 que en todos los restantes años, pero en el 2011 la proporción de puntuaciones bajas es mayor que en el 2009 y el 2010. Las puntuaciones en el grupo medio (entre 33% y 66%) son significativamente mayores en el 2009 y el 2010 que en el 2008 y el 2011. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 29 Figure 25: Distribución por año de los rangos que ocupan los estudiantes en MT En resumen, estos datos indican que en el 2011 se ha detenido el avance observado en los dos años precedentes, y sería interesante determinar cuáles factores están presentes en la implementación del Programa en ese año, para identificar las causas de esta detención, en relación con lo presentado en los resultados de Matemática. Figure 26: Distribución de rangos de puntuaciones en MT, por año Dado que, para Matemática, los grados 1º y 2º solamente se dispone de información de dos años, se realizó el análisis de la distribución por rangos para el 3º y 4º grados, solamente, que sí presenta la serie 2008 al 2011 completa. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 30 Table 26: rangos de puntuaciones por año, para 3º y 4º grados, Matemática Año de Evaluación 2008 2009 2010 2011 3ro Menos del 33% 45% 42% 42% 46% Entre 33% y 66% 49% 52% 53% 50% Más del 66% 5% 6% 5% 3% 4to Menos del 33% 29% 24% 25% 26% Entre 33% y 66% 65% 67% 66% 65% Más del 66% 6% 9% 9% 9% Leyenda: Naranja: Significativamente mayor que todos los que le siguen. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Verde: Significativamente mayor que 2008 y 2011. Morado: Significativamente mayor que 2009 y 2010, Amarillo: Significativamente mayor que 2011. Rojo: Significativamente mayor que 2008 Los grandes cambios se observan para 3º grado, donde los años 2009 y 2010 parecen ser los de mayor logro, con una caída o retorno en el 2011. La proporción de estudiantes que obtienen calificaciones por debajo del 33% se reduce significativamente en los años 22009 y 2010, pero regresa a valores semejantes a los del 2008 en el 2011. La proporción de casos superiores al 66% tiene su pico máximo en el 2009, luego se reduce hasta alcanzar niveles inferiores al 2008 en el 2011. Figure 27: Porcentajes de casos en los diferentes rangos de puntuación, en 3º grado, según año; para Matemática Para 4º grado, el 2008 es el año en que la mayor proporción de casos se queda con puntuaciones por debajo del 33%, mientras que luego de ese año la proporción de casos mayores a 66% es significativamente mayor que en el 2008. No se observan diferencias entre la distribución de proporciones en los años 2009 al 2011: No hay un efecto programa observable para esta variable. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 31 Figure 28: Porcentajes de casos en los diferentes rangos de puntuación, en 3º grado, según año; para Matemática 3.3.3 Un análisis de regresión simple, para MT También para MT se realizó un análisis de regresión lineal simple, con la introducción de Promedio en MT, como variable dependiente; y las variables Año de evaluación (2008 al 2011), Estrato (0=Rural, 1=Urbano), Sobre-edad (0=No, 1=Sí), Sexo (0 Femenino, 1 Masculino), Tanda (0=Matutina, 1=Vespertina) y Grado (años de escolaridad en Básica) como explicativas. Los principales resultados son semejantes a los observados en Comprensión Lectora, pero con un modelo menos explicativo: a) El modelo explica cerca del 11% de la varianza observada. b) Todas las variables introducidas se correlacionan con la variable dependiente (porcentaje de respuestas correctas en la prueba de MT), excepto la variable TANDA c) El modelo en sí es altamente significativo, con p<0.0005 d) Todas las variables introducidas resultaron ser significativas para explicar el modelo, incluso la variable TANDA. Table 27: Sumario de los modelos detectados para MT (Model Summaryb ) Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .113a .013 .013 .16814 .013 100.219 6 46878 .000 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 32 Los coeficientes del modelo se presentan en la siguiente tabla: Table 28: Coeficientes del análisis de regresión lineal en Matemática (Coefficientsa ) Model Unstandardized Coefficients Standardized Coefficients t Sig. 95.0% Confidence Interval for B B Std. Error Beta Lower Bound Upper Bound 1 (Constant) -12.257 1.574 -7.790 .000 -15.341 -9.173 Año de Evaluación .006 .001 .038 8.018 .000 .005 .008 Estrato (Dummy) .013 .002 .038 8.111 .000 .010 .016 Sobre-edad -.031 .002 -.061 -13.196 .000 -.035 -.026 Grado .015 .001 .054 11.771 .000 .012 .017 Sexo -.017 .002 -.050 -10.741 .000 -.020 -.014 Tanda (Dummy) .006 .002 .017 3.656 .000 .003 .009 a. Dependent Variable: Promedio Los coeficientes detallan la fórmula predictora de la puntuación real obtenida con base en las variables del modelo. A paridad de otros factores, se encuentra que: a) Por cada año del Programa hay una ganancia de un 1% en la calificación final b) Por estar en la escuela urbana se recibe un 1%. c) No estar en sobre-edad, significa una ventaja de un 3%. d) Cada año de escolaridad otorga un 2% e) Por el hecho de ser niña, se gana 2% en la calificación final. f) Los estudiantes de la tanda matutina ganan un 1% La fórmula predictora se convierte en: Pp = -12.257 + (0.006 * Año de Evaluación) + (0.013 * Estrato) - (0.031 * Sobre￾edad) + (0.015 * Grado) - (0.017 * Sexo) + (0.006 * Tanda). Traducido en un lenguaje figurativo, el perfil del estudiante más exitoso es semejante al ya descrito para CL: Una niña de cuarto grado matutino de una escuela urbana que está en la edad normativa para su grado, durante el año 2011; pero se sospecha que la relación con Año de Evaluación no es lineal, por lo que debe ser estudiado más a fondo. 4. Escuelas del Programa Versus Muestra de Comparación Los siguientes análisis reflejan los resultados comparativos entre escuelas del Programa PEF y la muestra de escuelas no en el programa (“Muestra de Comparación” o “MC”). Esta muestra solamente compara los grados 3º y 4º, para los años 2010 y 2011. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 33 Table 29: Proporción de sujetos en sobre-edad en la muestra control, por curso, área evaluada y año Año de Evaluación 2010 2011 Programa Comparación Programa Comparación Comprensión Lectora 3ro 0.52 0.47 0.52 0.50 4to 0.43 0.40 0.44 0.40 Matemática 3ro 0.37 0.34 0.36 0.33 4to 0.44 0.39 0.44 0.39 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, En todos los años y grados, la muestra de escuelas PEF obtienen resultados promedio superiores significativamente a los de la muestra de comparación. Figure 29: Escuelas PEF vs MC, por Área, grado y año Diferencias entre años, entre PEF y MC Table 30: Diferencias en los resultados entre años en PEF y MC, según área y grado (3º y 4º) Año de Evaluación 2010 2011 Comprensión Lectora 3ro PEF 0.52 0.52 MC 0.47 0.50 4to PEF 0.43 0.44 MC 0.40 0.40 Matemática 3ro PEF 0.37 0.36 MC 0.34 0.33 4to PEF 0.44 0.44 MC 0.39 0.39 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 34 Aunque hay diferencias significativas entre PEF y MC, el análisis de los últimos dos años no muestra grandes avances diferenciales: Para CL, tanto PEF en 4º grado como MC en 3º muestran resultados más altos en el 2011 que en el 2010; mientras en MT, los estudiantes de 3º grado del PEF tienen mejores resultados en el 2010 que en el 2011, manteniéndose inalterados los demás resultados. 4.1 Sexo y resultados en PEF y MC Los resultados muestran que, excepto en Matemática para la Muestra de Comparación, los resultados de las niñas son significativamente superiores a los resultados de los niños, en todos los años. Table 31: Resultados comparativos por sexo, con relación a la MC, por Área y año Niñas Niños 2010 Comprensión Lectora Programa 0.52 0.47 Comparación 0.43 0.40 Matemática Programa 0.37 0.34 Comparación 0.44 0.39 2011 Comprensión Lectora Programa 0.52 0.47 Comparación 0.43 0.40 Matemática Programa 0.37 0.34 Comparación 0.44 0.39 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 30: Resultados por sexo en PEF vs MC, según Área y año, 4.2 Sobre-edad y MC La comparación de estar o no en sobre-edad con las variables de año, grado, tipo de muestra (PEF o MC), muestra que los sujetos que no están en sobre-edad obtienen significativamente mayores promedios en las evaluaciones, excepto en el caso de la Muestra de Comparación en el año 2010 para MT en 3º grado, CL en 4º y en el año 2011 para MT en 3º grado. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 35 Table 32: Promedio en las pruebas, en casos con y sin sobre-edad, según tipo muestra (PEF o Comparasión), área, grado y año Sobre-edad No Sí 2010 3ro CL PEF .52 .48 MC .48 .42 MT PEF .37 .35 MC .34 .33 4to CL PEF .43 .37 MC .40 MT PEF .45 .40 MC .40 .34 2011 3ro CL PEF .53 .46 MC .51 .46 MT PEF .36 .33 MC .34 .31 4to CL PEF .45 .39 MC .41 .32 MT PEF .44 .41 MC .39 .34 Leyenda: Naranja: Significativamente mayor que la cifra que le queda a la derecha. Azul: Significativamente mayor que la cifra que le queda a la izquierda, Figure 31: Promedios comparados con MC, por sobre-edad, área, grado y año AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 36 APÉNDICE B-1 Tablas del Análisis de Regresión Lineal de Comprensión Lectora Regression Comprensión Lectora Descriptive Statistics Mean Std. Deviation N Promedio .4707 .18451 26000 Año de Evaluación 2010.28 .848 26000 Estrato (Dummy) .6113 .48747 26000 Sobre-edad .12 .321 26000 Grado 3.47 .636 26000 Sexo .52 .500 26000 Tanda (Dummy) .4904 .49992 26000 Correlations Promedio Año de Evaluación Estrato (Dummy) Sobre-edad Pearson Correlation Promedio 1.000 .163 .087 -.123 Año de Evaluación .163 1.000 .083 -.031 Estrato (Dummy) .087 .083 1.000 -.017 Sobre-edad -.123 -.031 -.017 1.000 Grado -.267 -.248 -.005 .054 Sexo -.150 .036 -.016 .122 Tanda (Dummy) -.054 -.083 -.088 .021 Sig. (1-tailed) Promedio . .000 .000 .000 Año de Evaluación .000 . .000 .000 Estrato (Dummy) .000 .000 . .003 Sobre-edad .000 .000 .003 . Grado .000 .000 .208 .000 Sexo .000 .000 .005 .000 Tanda (Dummy) .000 .000 .000 .000 N Promedio 26000 26000 26000 26000 Año de Evaluación 26000 26000 26000 26000 Estrato (Dummy) 26000 26000 26000 26000 Sobre-edad 26000 26000 26000 26000 Grado 26000 26000 26000 26000 Sexo 26000 26000 26000 26000 Tanda (Dummy) 26000 26000 26000 26000 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 37 Correlations Grado Sexo Tanda (Dummy) Pearson Correlation Promedio -.267 -.150 -.054 Año de Evaluación -.248 .036 -.083 Estrato (Dummy) -.005 -.016 -.088 Sobre-edad .054 .122 .021 Grado 1.000 -.016 .025 Sexo -.016 1.000 -.004 Tanda (Dummy) .025 -.004 1.000 Sig. (1-tailed) Promedio .000 .000 .000 Año de Evaluación .000 .000 .000 Estrato (Dummy) .208 .005 .000 Sobre-edad .000 .000 .000 Grado . .006 .000 Sexo .006 . .257 Tanda (Dummy) .000 .257 . N Promedio 26000 26000 26000 Año de Evaluación 26000 26000 26000 Estrato (Dummy) 26000 26000 26000 Sobre-edad 26000 26000 26000 Grado 26000 26000 26000 Sexo 26000 26000 26000 Tanda (Dummy) 26000 26000 26000 Variables Entered/Removedb Model Variables Entered Variables Removed Method 1 Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación . Enter a. All requested variables entered. b. Dependent Variable: Promedio Model Summaryb Model R R Square Adjusted R Square Std. Error of the Estimate 1 .346a .120 .120 .17311 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 38 Model Summaryb Model Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .120 590.892 6 25993 .000 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio ANOVAb Model Sum of Squares df Mean Square F Sig. 1 Regression 106.238 6 17.706 590.892 .000a Residual 778.892 25993 .030 Total 885.130 25999 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio Coefficientsa Model Unstandardized Coefficients B Std. Error 1 (Constant) -41.992 2.647 Año de Evaluación .021 .001 Estrato (Dummy) .027 .002 Sobre-edad -.050 .003 Grado -.069 .002 Sexo -.054 .002 Tanda (Dummy) -.012 .002 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 39 Coefficientsa Model Standardized Coefficients t Sig. 95.0% Confidence Interval for B Beta Lower Bound Upper Bound 1 (Constant) -15.862 .000 -47.180 -36.803 Año de Evaluación .098 16.148 .000 .019 .024 Estrato (Dummy) .071 12.044 .000 .022 .031 Sobre-edad -.087 -14.885 .000 -.057 -.044 Grado -.239 -39.750 .000 -.073 -.066 Sexo -.145 -24.755 .000 -.058 -.049 Tanda (Dummy) -.032 -5.492 .000 -.016 -.008 a. Dependent Variable: Promedio Residuals Statisticsa Minimum Maximum Mean Std. Deviation N Predicted Value .2929 .7073 .4707 .06392 26000 Residual -.64176 .66851 .00000 .17309 26000 Std. Predicted Value -2.781 3.701 .000 1.000 26000 Std. Residual -3.707 3.862 .000 1.000 26000 a. Dependent Variable: Promedio Charts AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 40 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 41 Regression Matemática Descriptive Statistics Mean Std. Deviation N Promedio .4051 .16920 46885 Año de Evaluación 2009.91 1.020 46885 Estrato (Dummy) .6164 .48627 46885 Sobre-edad .13 .336 46885 Grado 3.41 .626 46885 Sexo .50 .500 46885 Tanda (Dummy) .5295 .50011 46885 Correlations Promedio Año de Evaluación Estrato (Dummy) Sobre-edad Pearson Correlation Promedio 1.000 .038 .043 -.067 Año de Evaluación .038 1.000 .105 -.049 Estrato (Dummy) .043 .105 1.000 -.022 Sobre-edad -.067 -.049 -.022 1.000 Grado .047 -.046 -.010 .058 Sexo -.058 .024 -.041 .128 Tanda (Dummy) .003 -.199 -.077 .023 Sig. (1-tailed) Promedio . .000 .000 .000 Año de Evaluación .000 . .000 .000 Estrato (Dummy) .000 .000 . .000 Sobre-edad .000 .000 .000 . Grado .000 .000 .013 .000 Sexo .000 .000 .000 .000 Tanda (Dummy) .254 .000 .000 .000 N Promedio 46885 46885 46885 46885 Año de Evaluación 46885 46885 46885 46885 Estrato (Dummy) 46885 46885 46885 46885 Sobre-edad 46885 46885 46885 46885 Grado 46885 46885 46885 46885 Sexo 46885 46885 46885 46885 Tanda (Dummy) 46885 46885 46885 46885 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 42 Correlations Grado Sexo Tanda (Dummy) Pearson Correlation Promedio .047 -.058 .003 Año de Evaluación -.046 .024 -.199 Estrato (Dummy) -.010 -.041 -.077 Sobre-edad .058 .128 .023 Grado 1.000 .003 -.062 Sexo .003 1.000 -.021 Tanda (Dummy) -.062 -.021 1.000 Sig. (1-tailed) Promedio .000 .000 .254 Año de Evaluación .000 .000 .000 Estrato (Dummy) .013 .000 .000 Sobre-edad .000 .000 .000 Grado . .249 .000 Sexo .249 . .000 Tanda (Dummy) .000 .000 . N Promedio 46885 46885 46885 Año de Evaluación 46885 46885 46885 Estrato (Dummy) 46885 46885 46885 Sobre-edad 46885 46885 46885 Grado 46885 46885 46885 Sexo 46885 46885 46885 Tanda (Dummy) 46885 46885 46885 Variables Entered/Removedb Model Variables Entered Variables Removed Method 1 Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación . Enter a. All requested variables entered. b. Dependent Variable: Promedio AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 43 Model Summaryb Model R R Square Adjusted R Square Std. Error of the Estimate 1 .113a .013 .013 .16814 Model Summaryb Model Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 .013 100.219 6 46878 .000 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio ANOVAb Model Sum of Squares df Mean Square F Sig. 1 Regression 17.000 6 2.833 100.219 .000a Residual 1325.296 46878 .028 Total 1342.296 46884 a. Predictors: (Constant), Tanda (Dummy), Sexo, Grado, Estrato (Dummy), Sobre-edad, Año de Evaluación b. Dependent Variable: Promedio Coefficientsa Model Unstandardized Coefficients B Std. Error 1 (Constant) -12.257 1.574 Año de Evaluación .006 .001 Estrato (Dummy) .013 .002 Sobre-edad -.031 .002 Grado .015 .001 Sexo -.017 .002 Tanda (Dummy) .006 .002 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 44 Coefficientsa Model Standardized Coefficients t Sig. 95.0% Confidence Interval for B Beta Lower Bound Upper Bound 1 (Constant) -7.790 .000 -15.341 -9.173 Año de Evaluación .038 8.018 .000 .005 .008 Estrato (Dummy) .038 8.111 .000 .010 .016 Sobre-edad -.061 -13.196 .000 -.035 -.026 Grado .054 11.771 .000 .012 .017 Sexo -.050 -10.741 .000 -.020 -.014 Tanda (Dummy) .017 3.656 .000 .003 .009 a. Dependent Variable: Promedio Residuals Statisticsa Minimum Maximum Mean Std. Deviation N Predicted Value .3302 .4408 .4051 .01904 46885 Residual -.44084 .65343 .00000 .16813 46885 Std. Predicted Value -3.936 1.875 .000 1.000 46885 Std. Residual -2.622 3.886 .000 1.000 46885 a. Dependent Variable: Promedio Charts AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 45 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 46 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex B: Analysis of the ESP Database 47 APÉNDICE B-2: Bibliographia AMEX International (2012). USAID/ DR Evaluation Midterm Performance Evaluation, ppt presentation. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2010). Reporting Period: January 1, 2010- September 30, 2010. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2011). Reporting Period: January 1, 2011- May 31, 2011. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report. (2012a). Reporting Period: January 1, 2012- May 31, 2012. Effective Schools Program (USAID-PUCMM) Cooperating Agreement Report (2012b). Report October 1, 2010- December 31, 2011. Effective Schools Program (USAID-PUCMM) Safe Schools Program- Summary of Results. (2010). Reporting Period: October 1, 2009- June, 2010. Effective Schools Program (USAID-PUCMM) Safe Schools Program- Annual Report. (2010). Reporting Period: October 1, 2009- September 30, 2010. MINERD (2011). Politicas y Estrategias para la Ensenanza y el Aprendizaje de la Lectura y Escritura de Matematica desde los Primeros Grados de la Educacion Basica- Propuesta de Intervencion. Organización de Estados Iberoamericanos (2010). Republica Dominica hacia el 2021, Documento Del Pais. PEF (2010). Reporte de Resultados Primero y Segundo Grado, PEF Matemática 2010. PEF (2010). Reporte de Resultados Primero y Segundo Grado, PEF Lengua 2010. PEF (2011). Frecuencia de la Cobertura de los Contenidos de libro de Texto- tercero y cuarto [project generated materials figures] PEF (2012). Informe Anual Componente de Gestión y Participación (Julio 2011- Julio 2012). TEF (2009). Reporte resumen semestral Octubre- Diciembre 2008, Enero- Marzo 2009. TEF (2008). Reporte trimestral: Julio- Septiembre, 2008. USAID (2009). Attachment B Program Description for Effective Schools Program. Dominican Republic. USAID (2010). Performance Monitoring & Evaluation- TIPS Constructing an Evaluation Report, No. 17, Washington, D.C. USAID (2011). USAID Education Strategy- Education Opportunity Through Learning, Washington, D.C. Valverde, G. & Naslund-Hadley, E. (2010). The State of Numeracy Education in Latin America and the Caribbean, Inter￾American Development Bank, Washington D.C. AID517-TO-12-00001 USAID/DR Education Portfolio Evaluation Annex C: USAID/DR Basic Education Program (BEP) Mid￾Term Performance Evaluation Submitted to: USAID/Dominican Republic Prepared by: Tonya Giannoni, Ph.d., Evaluation Specialist, BEP Edited by: Virginia Lambert, Team Leader for Portfolio Evaluation January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program i Basic Education Program (BEP) Mid-Term Performance Evaluation Annex C: Table of Contents 1. Introduction ............................................................................................................................................. 1 1.1. Description of the Program Evaluated............................................................................................... 1 1.2. Evaluation Purpose and Methodology............................................................................................... 2 2. Findings and Conclusions ..................................................................................................................... 4 2.1. Performance ...................................................................................................................................... 4 2.2. Efficiency .........................................................................................................................................10 2.3. Sustainability....................................................................................................................................11 3. Summary of Recommendations ..........................................................................................................12 Appendices................................................................................................................................................17 Appendix C-1: Descriptive Data about BEP Interventions .....................................................................17 Appendix C-2: Schools Visit and Classroom Observations....................................................................19 Appendix C-3: EpC Programmatic Data................................................................................................. 21 Appendix C-4: Documents Reviewed.....................................................................................................24 Bibliography ..............................................................................................................................................24 List of Tables and Figures Table A: Mechanisms Supporting Schools ................................................................................................... 1 Figure A: Theory of Change for the Basic Education Program..................................................................... 2 Table B: Classrooms Observed .................................................................................................................... 3 Figure B: BEP at the Mid-Term Evaluation................................................................................................. 13 Table 1: Number of Schools (total =51) that received the intervention.......................................................17 Table 2: Distribution of Schools by Geographic Area .................................................................................17 Table 3: Distribution of Schools by Sponsor (as of March 2012)................................................................18 Table 4: Available Data on Teachers and Directors Trained by Ediciones SM ..........................................18 Table 5: Schools Visited by Evaluation Team and Interviews Conducted..................................................19 Tables 6: Distribution of Classrooms Observed..........................................................................................19 Tables 7: Classroom Size (Attendance) and Teacher Years of Experience ..............................................19 Tables 8: Classroom Observations: Adoption of Learning Strategies ........................................................20 Figure 1: Classroom Observations (n=15)..................................................................................................20 Table 9: Children Served in EpCs (2011-2012) by EpC Name, School, and Sponsor...............................21 Table 10: Children Served in EpCs (Period Unknown)...............................................................................21 Table 11: Percentage of Girls and Boys Served in EpCs (Period Unknown).............................................22 Table 12: Reason for Participation: Opportunity or Overage......................................................................22 Table 13: Percentage Distribution of Problem Area for Students by EpC ..................................................23 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 1 1. Introduction This report presents the major findings and conclusions of the mid-term evaluation of the Basic Education Program (BEP) implemented by the American Chamber of Commerce in the Dominican Republic (AMCHAM/DR) and funded by the United States Agency for International Development (USAID) and Dominican private sector partners. In addition, the evaluation team has developed a set of recommendations to strengthen the implementation of the program and the sustainability of the USAID investment. The recommendations are provided for USAID and the implementing partner to consider as course corrections as well as for the future of AMCHAM/DR's school sponsorship activities. The conclusions and recommendations contribute to the overall integrated performance evaluation of the USAID/Dominican Republic Portfolio evaluation. 1.1. Description of the Program Evaluated The Basic Education Program is a public-private partnership among three types of organizations with the common goal to “contribute to improving the quality of basic education in public schools in the Dominican Republic (DR) by promoting an active involvement of the private sector” (USAID 2009, 10). Table A summarizes the three mechanisms that together support the partnership for working at the school-level: Table A: Mechanisms Supporting Schools Agreement Parties Period of Performance Dollar Value Cooperative Agreement Number 517-A￾00-10-00100-00 United States Agency for International Development in the Dominican Republic (USAID/DR) and AMCHAMDR October 1, 2009 to September 30, 2014 $2,303,489.14 (USAID) $1,373,453.11 (Cost-Share) $3,676,942.25 (Total) Memorandums of Understanding Existing private sector partners with active agreements include: Varies Annual budgets determined with sponsor Alliance agreement Ministry of Education, USAID and AMCHAM/DR Unknown Not specified. This evaluation is focused on the Cooperative Agreement (CA). As outlined in the CA, the purpose of the AMCHAM/DR agreement with USAID is to improve the quality of basic education in 139 public schools through “comprehensive, coordinated, and systematic interventions” led by AMCHAMDR and implemented through “strategic alliances between the public and private sectors with AMCHARMDR administrative and technical assistance” (p. 10). The results framework for BEP is as follows: Component 1: Private Sector Involvement Result 1: Business Involvement Increased (target of 125 AMCHAMDR business members) Results 2: Comprehensive, sustainable and integrated interventions developed and implemented (number of plans developed and implemented) Component 2: School Sponsorship Interventions (target 139 schools) Result 1: Teacher pedagogical skills improved (targets are 80% of participating students show improved grades and 80% of participating teachers utilize new methodologies and teaching techniques) Result 2: Schools have relevant didactic material and tools to support pedagogical interventions Component 3: Strategic Alliances with the Private and Public Sectors Result 1: Public-Private Strategic alliances established (interventions to adjust and/or upgrade school’s infrastructure) AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 2 Result 2: A stronger and extended Corporate Social Responsibility (CRS) Committee Functioning (numbers of meetings and programs). Component 4: Assessment, Monitoring and Evaluation Result 1: An outreach and communication plan designed and launched to promote the results of the sponsored interventions The cooperative agreement identifies a series of packages of interventions that served as a menu for the school-level interventions. These include: o Teacher training to improve reading instruction and pedagogy (utilizing various vendors). o Provision of didactic materials, library and reading corner materials, and/or other types of equipment for the school. o After-school or non-formal education programs, including Espacios para Crecer (EpCs). The basic theory of change for BEP is presented in Figure A. The four components of BEP together will provide the outcome of improved educational achievement. The interventions are defined prior to the start of each academic year. BEP funding under the existing cooperative agreement ends in September 2014. As a result, BEP (as designed) will have one more academic year to implement any changes (2013-2014). 1.2. Evaluation Purpose and Methodology This mid-term performance evaluation was conducted as part of an overall USAID/DR Education Portfolio Evaluation. The evaluation team utilized standardized evaluation instruments (key informant interviews, classroom observations, teacher and director interviews, and document review) to collect the data utilized in the evaluation. For a detailed description of the methodology employed in the USAID/DR Education Improvements in Educational Achievement Private Sector Sponsor School-Based Interventions Alliances & Monitoring of Progress Figure A: Theory of Change for the Basic Education Program AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 3 Portfolio Evaluation see the USAID/DR Education Portfolio Evaluation Integrated Report (Integrated Report). The field work for the evaluation was completed in October 2012. To ascertain if BEP is on track to meet its targets, data collected specific to this evaluation included school-visits and classroom observation. The data was collected to respond to evaluation questions about performance, which examine which program components are being implemented, how they are being implemented, and if they are being implemented according to plan. The evaluation team integrated gender considerations by analyzing the extent to which gender analysis has been utilized in the design and implementation of the various interventions and if there are significant differences in the level of participation of either girls or boys in the classrooms. During the document review, the evaluation team found that for BEP there is a variation in the intensity of interventions (number and types) by school. Based on this preliminary finding, the evaluation team proposed to select schools for site visits based on the number of interventions being implemented in the school. Selecting the schools in this way allowed the team to be able to conduct classroom observations in a variety of settings. As part of the planning process, both USAID and the evaluation team agreed that the intensity of the interventions may affect perception of the BEP program at the school level. Table B summarizes the distribution of the schools by intensity of intervention. Utilizing AMCHAMDR records (list of 51 schools as of 3/30/2012), the team selected schools for classroom observation. The selection process was as follows: o Schools that were outside of a geographic focus area for USAID were removed from the list (43 schools remained). o For each type of intervention, the school was scored as 0 or 1. o The number of the interventions was summed by school and then grouped or stratified (column 1). o A software program (MS Excel Data Analysis Tools) was utilized to generate a random selection (column 2) for each grouping. The evaluation team scheduled visits to a total of 11 schools although one of the schools was closed due to an unexpected holiday (celebrations of the Teacher Union’s elections). The distribution of the schools visited (10) and classrooms (15) observed varied from the original design selection because the actual interventions received differed from the records reviewed. The gaps in records and timing are examined in the next section. Table B: Classrooms Observed Column 1 Total Number of School per AMCHAMDR Records Column 2 Selected Schools Column 3 Schools/Classrooms Visited Four/Five Interventions 11 3 4/4 Three Interventions 10 3 4/8 Two Interventions 12 4 1/1 One or Zero Interventions 10 2 1/2 43 12 10/15 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 4 As the evaluation questions (see the Integrated Report) were finalized, in consultation with USAID, it was determined that two elements of the BEP program would be excluded from the evaluation. These are: • Recruitment of Private Sector Sponsors In the near term, AMCHAM/DR is conducting an assessment of the private sector partnership component; therefore, this evaluation does not assess the performance targets related to securing the number of partners outlined in the program description. • Educational Achievement. USAID is financing the application of a standardized measurement of student achievement (Early Grade Reading Measurement and Early Grade Mathematics Measurement) that was underway at the time of the field work for this evaluation. Thus, understanding performance for this element of the theory of change is not part of the evaluation. 2. Findings and Conclusions The findings and conclusions are presented by major question area: performance, efficiency, and sustainability. 2.1. Performance The portfolio review employs the following definition of performance; “the contribution of each component to quality basic education, benefiting girls and boys equally” (USAID Contract with AMEX International 2012). Guiding questions include: Which components are being implemented and where?, How are they being implemented?, What are the characteristics?, Who is benefiting and to what degree?, and Is the project on track to meeting targets? Measuring performance and attributing interventions to BEP’s actions at the school level faces several contextual and programmatic constraints. Context: In the Dominican Republic, there are many institutions (governmental and non￾governmental) that intervene at the school level and the coordination among these institutions is at times weak. The Ministry of Education (MINERD) has divided the country into three regions to support basic education services via strategies in teacher training in mathematics and literacy. These organizations are Pontificia Universidad Católica Madre y Maestra (PUCMM)1 in the north, which is supported by USAID; the Cultural Center POVEDA in the southeast, and the Organization of Ibero-American States (OEI)2 in the east (Basic Education Program Dec 2011). Non-governmental interventions include Junior Achievement and other donor-funded activities. Program Specific: There is no baseline in the BEP schools for either student or teacher achievement or analysis of their perceptions from the start of this cooperative agreement.3 In addition, many of the schools have been receiving assistance from the sponsors and AMCHAM/DR prior to the initiation of this cooperative agreement. For example, 35 school files were selected for detailed review during the evaluation. Sixteen began working with AMCHAM/DR before the signing of this cooperative agreement usually as of 2007 (of the 10 schools that were visited, seven school were legacy schools). 1 BEP has signed one agreement with PUCMM for work in one school. 2 BEP received a proposal from OEI to develop murals in 31 BEP schools across geographic areas. 3 A baseline is being completed now and offers a potential for an assessment of early grade reading and math skills at the program conclusion. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 5 A second program-specific constraint is that BEP records have not been systematically and rigorously updated regularly for each participating school and there do not appear to be files kept for directors or teachers. After the initial intake assessment, there is no formal detailed written assessment in the files. The evaluation team received a list of schools and identified the package of interventions that were being implemented in those schools (Basic Education Program March 2012). However, these records did not always match the vendor reports when there were reports (PRISMA Febrero 2012). In other cases, the reports were missing and while data was sought during the evaluation, there is an incomplete record of the interventions by school and during what period. For example, during the field work data collection period (October 2012), the evaluation team was only able to verify that Quantum Learning training had occurred but could not prepare tables to understand what schools or teachers participated. The evaluation team has no evidence and no expectation that the interventions reported have not occurred, but because BEP does not provide explicit and specific reporting requirements for its vendors the data that are reported do not match BEP files. An example of this is the reporting of data by Entrena for Quantum Learning or EDUCA for the EpCs (see below and Appendix C-3, Table 9 for an example). The effect is discussed further under Finding #1 below. Based on these contextual and program-specific constraints, the evaluation team focused on attempting to ascertain if teachers are adopting new methodologies and if they perceive BEP as a stimulus for the change. The evaluation team had to dedicate a significant amount of time to gather basic information about what interventions had occurred at the school-level and by beneficiary type (director and teacher). The data that were collected by the evaluation team is summarized in Appendix C-1.4 Utilizing the available information and the methods as outlined in Section 2 of this annex, the evaluation team developed four findings related to performance focused on answering the evaluation questions: (1) which project components are being implemented?; (2) where, and how are they being implemented?; (3) who is benefiting (disaggregated by sex, region, position, to the extent possible), and to what degree?; and, (4) is the program on track for meeting the targets identified in the PMP? Finding #1: The program goals and components are generally clear (and similar) for AMCHAMDR and for the Sponsor, but not necessarily for the beneficiaries (directors and teachers). Conclusion #1: BEP has yet to achieve a critical benchmark as outlined in the program purpose: a “comprehensive, coordinated, and systematic intervention” plan. AMCHAM’s role at the school level is undifferentiated and attribution of AMCHAM assistance is (and will likely always be) un-attributable. During interviews and in written documents, private sector sponsors and the program implementers (AMCHAM/DR and its vendors) all summarize the program goals and objectives similarly. To date, BEP has generally met the targets for raising matching funds for each school. In addition, the interviews conducted by the evaluation team confirm that stakeholders agree that AMCHAM/DR is the correct entity to provide administrative and technical assistance for a basic education program funded by private sector sponsors. This was the case even though BEP donor stakeholders acknowledged that the reports and data available about the implementation of BEP are insufficient. BEP and its donors are supporting a lot of good deeds and actions, but due to the lack of a systematic reporting system and analysis the evaluation team can reflect on good deeds but not good outcomes. 4 The original files are available upon request. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 6 In contrast, the role of AMCHAM/DR is less clear to stakeholders at the school-level. The indirect beneficiaries (directors and teachers) could not always differentiate AMCHAM (or their sponsor) from assistance being provided by others (e.g., Ministry or Poveda). All of the directors interviewed had attended the AMCHAM/DR sponsored training in August 2012. When asked to name institutions that had provided them training during 2012, the majority of the directors mentioned AMCHAM and the sponsor amongst several actors.5 Yet, even when prompted specifically about AMCHAM/DR, two directors did not name a specific program offered. The teachers were less likely than the directors to recognize AMCHAM/DR or the sponsor by name when asked an open-ended question. Teachers could discuss methodologies that they had been taught but did not associate this them with an integrated program or AMCHAM/DR. The more likely responses were Poveda, the Ministry, and at times SM (the organization that conducted the training). Those that the evaluation team believed had received training in Quantum Learning for Teachers generally could not name the training, and this type of training did not appear to be linked to AMCHAM/DR or the sponsor. When specifically asked about AMCHAM/DR and follow-up, most directors could name the BEP’s education coordinator. However, they were not all clear about the process for making decisions about the types of support to be provided. For infrastructure improvements, the directors were more likely to talk specifically about the engagement with the private sector sponsor. In addition, although AMCHAM/DR developed a menu of types of interventions to be implemented in schools, there was not clear evidence of coordination among the various types of interventions. For instance, it was not clear that there was a relationship between AMCHAM/DR and incorporation of non￾formal education programs (“Espacios para Crecer,” EpCs). The evaluators could only obtain tentative answers about EpCs, Quantum Learning for Teachers or other training or after-school interventions that had been funded by AMCHAM/DR. It was very difficult to evaluate the EpC program because the EpC program had effectively been suspended for the new school year by AMCHAM/DR. Based on the available data (the vendor’s records and those submitted to AMCHAM/DR), BEP had supported a total of 19 schools (with EpCs) since 2009. In 2009, only Fundación Rica participated. By the third year (school year 2011-2012), five different companies supported EpCs. The EpCs were not closely assigned to schools by AMCHAM/DR. Records kept by the vendor do not align exactly with the schools supported. For example, in Constanza, seven communities were supported. Their alignment to specific schools is unclear in the monitoring records. The records available from the vendor suggest that EpCs have helped overage children primarily in both reading and math or in reading only. At the time of the evaluation, the data was incomplete. Excluding non-reporting centers, more than 60% of those attending EpCs passed reading and/or math, but there is no control group to allow attribution directly to the EpCs. Available data are included in Appendix C-3. In one case, the school director invited the evaluation team to visit the EpC suppored by the school. However the EpC was in name only and did not include any of the programmatic traditions of the EpC. Similarly, although there was some evidence in classroom observations that teachers were utilizing techniques from Quantum Learning (another type of intervention), there was no linkage od this training to BEP, AMCHAM/DR, or the sponsor. 5 All of the Directors received an introductory letter from the Ministry that could have been a prompt. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 7 This lack of attribution when compared to the interviews at the school-level for ESP was striking to the evaluation team. The implication or conclusion is that while BEP may now use a training plan agreed by the school, and between BEP and the sponsor, the legacy of the ad-hoc interventions of the early years of the cooperative agreement remain. This lack of congruence between plans and observations has important potential consequences for the model. All of the sponsors interviewed (representing 32 of the 51 schools included in the evaluation) stated that working through AMCHAMDR was important because of AMCHAM/DR’s knowledge of education. The value-added proposition of AMCHAMDR for the sponsor is the ability to develop, implement, and monitor a “comprehensive, coordinated, and systematic” intervention to improve the quality of basic education in the public schools of the Dominican Republic.” Attribution ,even when based on perceptions, is important for this type of model. Finding #2: There are inconsistencies and missing data among BEP’s records. Conclusion #1: BEPs written records are insufficient for effective monitoring of the M&E Plan or to achieve the Result No. 1 under Component No. 4. As outlined above, the quality of data available to the evaluator is insufficient to answer every question included in the evaluation design. This finding is not a surprise and is recognized by all parties involved in BEP. AMCHAM/DR is currently taking steps to strengthen performance monitoring. Some elements that are unclear at this time include: • The number of teachers trained, disaggregated by school and intervention type (a total is included in the quarterly report) and by year, and if those teachers remain in the schools being assisted. • Exact timing of interventions in each school. • Extent to which the training has changed teachers' actions in the classroom. • Number of students (disaggregated by school), who benefit from USAID’s and the sponsor’s financial support (there are numbers reported to sponsors). The implications of this finding are related to Component 4 as well as the overall success of the program. Without clear and verifiable basic information at the school and individual level, BEP will be unable to measure its importance for the private sector sponsors or for its donors. While every teacher and director cited that they perceived that training was producing change, there was only partial evidence in the classroom (see Appendix C-2). According to BEP’s new plan (Basic Education Program 2012) and key informant interviews, during the next year BEP will be undertaking actions that are important to strengthening Component 4: • Collect student performance data (math and reading) and promotion rates (Basic Education Program 2012). • Monitor the time that teachers dedicate to reading in the classroom (Basic Education Program 2012). • Receive from EDUCA (a vendor) a report on the effectiveness of the after-school intervention— Espacios para Crecer—on addressing the challenges of overage students in math and reading (Interviews 2012). • Add a staff member dedicated to monitoring and evaluation (Interviews 2012). Finding #3: BEP’s interventions utilize commercially available training programs for improving teacher’s performance. Conclusion #3: While BEP uses standard methods to train teachers to improve pedagogy, these generic methods rely heavily on strong program management. There are two elements of this finding: AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 8 a) BEP/AMCHAM/DR is providing commercially available training programs. Rather than developing its own teacher training program, AMCHAM/DR has gone out to the marketplace to solicit proposals for delivering teacher training in the sponsored schools. In the DR several methods for how to organize a classroom and the use of participatory methods in the classroom have become standardized. For example, site visits demonstrated that teachers in BEP schools decorated their walls in a similar way to teachers in classrooms where other USAID-financed programs (e.g., ESP, the Batey Community Development program) were being implemented. They also introduced learning corners with a similar frequency, and attempted to introduce classroom libraries (see frequency tables in Appendix C-2). In addition, in BEP schools, teachers elicited student participation (by asking questions) in more than 70% of the classroom observations. Classroom observations demonstrated that students participated in class (see Appendix C-2). Engaging students to participate individually is the first-step in a shift to a more active learning method (Questions 13B3/B4). However, the use of more advanced techniques, such as having the teacher demonstrate physically or with materials, or having students role play were used very infrequently (0% to 20%), see Figure 1 in Appendix C-2. Based on the teacher interviews and classroom observations, teachers recognized some of these innovative methodologies as effective and nearly universally expressed interest in using more participatory teaching methods. b) BEP/AMCHAM/DR does not have a unique brand or provide a systematic program. AMCHAM/DR has outsourced its technical assistance to a series of vendors. While nearly all directors could name the education coordinator for AMCHAM/DR (confirming that monitoring visits are occurring), the directors could not describe in detail how decisions about what type of support they are receiving are made or summarize the general purpose of the assistance. It is clear that the directors are grateful for the support, but when compared to the interviews conducted for the Effective Schools Program (ESP) there was significantly less recognition that they were being provided an integrated, results-driven program. There are potential positive and negative consequences of this finding. A positive implication is that since BEP is not recreating the wheel there is the possibility that its interventions can be better coordinated and can avoid sending conflicting messages to teachers and directors. There were only a very few instances when teachers and directors reported incidents of when they were receiving conflicting messages from the Ministry and the workshops that they attended. The challenge for this approach is that AMCHAM/DR must have a strong programmatic design that is systematic and a priority, and coordinated to assure that the disparate programs it procures are an integrated program that is greater than the individual interventions. Finding #4: The institutional environment at the school level is complex with multiple actors. Conclusion #4: The implementing environment is a challenge for the structure of BEP. To date, AMCHAMDR has yet to define a clear mechanism for ensuring coordination at the school level. Coordination is a key factor for the performance success of BEP. BEP is seeking to increase the number of sponsors and the number of schools that benefit from a public-private partnership. AMCHAM/DR as the entity providing the technical assistance and administrative support must coordinate its activities with a number of partners. Coordination must occur with the following partners: • Ministry of Education (Central, Regional, and District) • Other sponsors (at the school-level) AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 9 • Ministry of Education assigned-TA providers (POVEDA, PUCMM, and OEI) • The municipality • Other NGOs working in schools/communities • Parents’ associations There are mixed-results at each institutional level of coordination: School-Level. Based on the interviews at the school-level, there was minimal coordination with other actors engaging at this level (POVEDA, municipality, and even the sponsor). Directors and teachers are not able to express in a dialogue how these entities are or are not coordinated. In some cases, they were concerned about getting too much advice or directions. Another challenge for coordination in the implementing environment is that sponsors engage directly with the schools on activities “unrelated” to BEP. For example, AES-Dominicana is providing Junior Achievement support. The evaluation team could not, based on the interviews and site visits, see how these different types of support are coordinated. The proposals made by BEP to the sponsors for the 2012 school-year indicated that director and teacher training programs (Basic Education Program June 2012) would be coordinated with the appropriate entity, but it seems that SM (the selected vendor) trained all of the directors. Sponsor-Level. Based on the data available at the time of the evaluation, AMCHAM/DR has regularly prepared reports for the sponsors. The reports give details about what has been done, but it seems that sponsors make decisions about what is going on in the school based on their own visits rather than AMCHAM/DR reports. For example, one sponsor indicated that a school that it is sponsoring was given a specific list of criteria for the school to continue to receive support.6 Despite this very important decision and analysis (based on the lack of progress), AMCHAM/DR did not report this information in any of its monitoring reports or in its quarterly reports to USAID. Another aspect of coordination is about coordination by and between the sponsors. There have been limited opportunities for the sponsors to interact about BEP. An important negative consequence of this lack of sharing is that the model for intervening at the school-level (i.e. contracting vendors) and the appropriate coordination mechanism with the Ministry (e.g. AMCHAM/DR or the private sector sponsor) have not been discussed to identify key lessons learned or to test the assumptions about the model. Challenges (such as coordinating all interventions by sponsors at the school level) are also not discussed. The operating principles or actions are not reviewed across the sponsors. As far as the evaluation team could ascertain, there has been one meeting (outsourced to EDUCA) that had some attendance of sponsors (Basic Education Program Jan-March 2012). The agenda was more broadly focused on basic education and was not an opportunity to discuss school sponsorship under BEP. BEP has sponsored trips to other countries but this activity was not mentioned by the sponsors. Ministerial-Level. AMCHAM/DR has worked to engage with the Ministry of Education at an operational level. For example: • There is evidence from the key interviews and Finding #3 (what is observed in the classroom and the teacher interviews) that the vendors coordinate their products with the Ministry at an operational level (especially related to the Escuela de Directores). • BEP documentation suggests that there is a sufficient-level of coordination between the Ministry, AMCHAM/DR, and the school-sponsors for infrastructure repairs (Basic Education Program, 2010). 6 The name of the school and sponsor is excluded from the public report. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 10 • Interviews with Ministry personnel suggest that there is no coordination between AMCHAM/DR and the curriculum and instruction departments in the Ministry, although in the past there was a relationship with the department responsible for community outreach. The implication of this performance is that the lack of coordination reduces the overall efficiency of the dollars being invested in the public schools. There are additional implications that affect sustainability, as outlined below. 2.2. Efficiency This section of the evaluation answers the question: what are the aspects of the projects that are operating more or less efficiently and why? There are four inter-related findings and conclusions about the efficiency of BEP implementation. Finding #5: Purchasing books and donated materials is the easiest means for sponsors to make a financial contribution. Conclusion #5: While an efficient means to move funds, the focus on donation has dominated the engagement with the school and it is unclear if the per student cost7 for BEP is efficient. Finding #6: BEP organizes its records by contract (vendor) to meet grant reporting requirements. Conclusion #6: BEP’s management organization for its files obfuscates what is happening at the school-level. Finding #7: BEP has efficiently contracted various types of interventions (to form a package). Conclusion #7: The interventions have been defined and implemented but without an analysis of their results (to date). Finding #8: Teachers expressed that instruction in reading was their greatest content challenge. Conclusion #8: Most (but not all) of BEP’s attention is focused on addressing what teachers cite as the methodology and area in which they need assistance. AMCHAM/DR has focused its management of BEP to be efficient. This efficiency is written into the program description for BEP. There are several examples of how AMCHAM/DR has achieved efficiency. • AMCHAM/DR has focused its teacher training on the most-cited problem of teachers (reading). • AMCHAM/DR has organized its operations around the principles of outsourcing and financial management. Files are structured to respond to budgetary and financial management questions. This is efficient for the management for the cooperative agreement, but obfuscates programmatic management. • AMCHAM/DR has a very small staff with the director, one programmatic professional (education coordinator), an administrative/financial staff member, and a driver. • According to key informant interviews, Result 2 under Component 2 (donation of materials) is the most efficient type of intervention (and the most utilized). Efficiency is a valid metric. However, efficiency for what is the key question. The focus on efficiency has diminished the AMCHAM/DR value proposition in two ways: • AMCHAM/DR has become very efficient at a task for which it is very difficult to differentiate itself. There are many actors that make donations to schools. Furthermore, as discussed with school 7 This cannot be measured at this time due to the lack of data. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 11 directors and sponsors, the private sector sponsor gives materials and makes infrastructure updates directly (without the need for the intermediary). • AMCHAM/DR has made decisions about the type of interventions based on administrative costs versus best value for the private sector sponsor and USAID investment. The clearest example of this is the decision to not utilize the PUCMM model for improving learning outcomes in reading or better coordinating BEP's interventions with POVEDA. The BEP interventions have been efficient, but the question remains: is AMCHAM/DR adding value? 2.3. Sustainability The evaluation team has approached the question of sustainability on two tracks. The first is to determine if program interventions are likely to be sustained after USAID’s involvement in the program. The second is to ascertain if the package of interventions produce sustainable change in learning outcomes. In the case of BEP it was not possible to develop a clear understanding of the outcomes of the package of interventions (see Finding #1). For the current school year, AMCHAM/DR has developed a comprehensive package. The question of sustainability of the interventions at the beneficiary level will be best answered during the final evaluation. Otherwise, there are three inter-related findings and conclusions about the question of sustainability. Finding #9: School-sponsorship and AMCHAM/DR’s role is perceived as valuable by the private sector. Conclusion #9: Private sector involvement as a sponsor and working through AMCHAM is on track to be sustainable. Finding #10: The Ministry, AMCHAM/DR, and the sponsors have different definitions of a sustainable public-private strategic alliance for basic education. Conclusion #10: Without a clear strategy for building a strategic alliance for intervention at the school-level, BEP may sustain interventions without achieving improvements in education. Finding #11: Schools have relevant didactic materials but it is an area of concern; materials exist but their use and incorporation into teaching is uneven (based on the limited observations). Conclusion #11: Access to materials (equipment and didactic tools) and training of teachers are necessary but not sufficient condition to change behavior. The private sector partners are universal in their support for an entity to operate educational programs utilizing donated funds to meet their particular corporate social responsibility objectives. Thus, a key variable for the sustainability of this type of initiative exists. However, transforming this general interest on the part of the private sector into a sustainable public-private partnership that improves educational quality is not assured. The program documentation available and interviews suggest that the strategy for private sector engagement is to strengthen the direct links between the private sector and the Ministry. BEP has, to a certain extent, outsourced the building of this strategic alliance to EDUCA. However, the EDUCA agenda (advocacy for education) is distinct from sponsorship and the dialogue that is necessary between the sponsor, AMCHAM/DR and the school. Interviews with individuals representing the Ministry suggest that the role of the private sector in basic education—with a focus on improving educational attainment is not clear. Furthermore, it is not clear from the team’s discussions with the school directors if the interventions as developed and implemented by AMCHAM/DR are sustainable in the school. Interviews with school directors suggest that they are very open to assistance. However, there is a disconnect between their AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 12 interest in the support and their understanding that there is a comprehensive program. For instance, after nearly two years of programming AMCHAM/DR set-out to develop a general proposal for basic education. This proposal was developed in writing and presented to the sponsors as a means to gather the sponsor’s support. However, it is unclear if this analysis and proposal has been developed with the directors. As noted above, an understanding of BEP's goals is not clear among all directors. AMCHAM/DR is known as an entity at the school-level (among directors), but there is no recognition of BEP as a program. The directors can discuss the particular interventions but not a comprehensive program. The implication of this finding is that while private sector partners may continue to sponsor schools and want AMCHAM/DR to administer the programs, the theory of change that sponsorship (i.e. money) will produce improvements in education attainment is not assured. It is just as plausible that money will provide new materials and change teaching instruction, but learning will not improve. Many sponsors have been working for more than five years with schools. They see improvements in the infrastructure. Directors and teachers say things are better. However, there is no evidence base to support this. What is missing? A strong link between money and outcomes is necessary. Addressing some of the weaknesses (such as coordination and selection of interventions) may be the first step. 3. Summary of Recommendations The recommendations developed through the course of this evaluation are designed to be implementable and are focused on answering one of the key evaluation questions: How can the design, management and execution become more efficient in achieving program goals? The recommendations flow from the respective conclusions. The basic design of BEP is sufficiently clear and defined as illustrated in the theory of change (see Figure 1). However, as outlined in the evaluation the theory of change requires coordination and integration. As BEP is currently being implemented (see Figure B), AMCHAM/DR has a strong relationship with the sponsor and USAID (solid arrow), a weaker relationship with the school (lightly shaded), and a limited relationship with MINERD and other alliances (outlined arrow). This is a very different structure from the ideal (theory of change) implied in the design. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 13 Figure B: BEP at the Mid-Term Evaluation Theory of Change As Implemented This evaluation is one element of the current review of the USAID Education Portfolio and BEP. Consequently, the recommendations should be reviewed closely once the results of EGRA/EGMA are received and the assessment of what motivates private sector sponsorship of schools is complete. Once USAID has the results of these two other tools, the evaluation team recommends that USAID address the following three questions: • Should the targets for the number of private sector partners be decreased? • What is the right mix of donations (materials) and technical assistance for BEP? • Should the programmatic purpose be updated to reflect a niche strategy for private sector sponsorship? Based on the information collected during the evaluation, the evaluation team has four recommendations outlined below. AMCHAM/DR School Sponsor USAID MINER D Alliance s Improvements in Educational Achievement Private Sector Sponsor School-Based Interventions Alliances & Monitoring of Progress AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 14 Recommendation #1: BEP should improve its coordination of its activities at the school-level with the various organizations working in each school, build its relationship with the Ministry, brand and focus its interventions, and measure results. To address the following conclusions: Conclusion #1: BEP has yet to achieve a critical benchmark as outlined in the program purpose: a “comprehensive, coordinated, and systematic intervention” plan. AMCHAM’s role at the school level is undifferentiated and attribution of AMCHAM assistance is (and will likely always be) un￾attributable. Conclusion #3: While BEP’s interventions are complementary to existing standards they are generic and dependent on strong programmatic management in order to have much effect on learning. Conclusion #4: The implementing environment is a challenge for the structure of BEP. To date, AMCHAMDR has yet to define a clear mechanism for ensuring coordination at the school level. Conclusion #8: While an efficient means to move funds, the focus on donation has dominated the engagement with the school and it is unclear if the per student cost8 for BEP is efficient. Conclusion #10: The interventions have been defined and implemented but without an analysis of their results (to date). Conclusion #11: Most (but not all) of BEP’s attention is focused on addressing what teacher’s cite as the methodology and area in which they need assistance. Improve coordination. For the current fiscal year, AMCHAM/DR should clarify and improve the coordination at the school-level through dialogue with the school directors, associated stakeholders (other technical assistance providers designated by the Ministry), and the school’s sponsor. This is important because for the current school year the plan to utilize an outsourced vendor (Ediciones SM) continues. USAID might be in a position to facilitate this process at least among the various interventions it funds. The coordination objectives as outlined in the reformulated plan for assistance presented to sponsors (December 2011 letter to a sponsor) remain important. If the sponsors have not already been updated on the revision (moving away from utilizing PUCMM, etc) they should be. In addition, the signing of the MOU is an important step for coordination but the draft version reviewed as part of the evaluation does not describe the specific interventions (Unsigned Tripartite MOU – Escuela KM). Build relationships. BEP will benefit from a direct and clear relationship with the Ministry. As outlined above, sponsors seem to want AMCHAM/DR to do this and AMCHAM/DR seems to want to facilitate the private sector establishing a direct relationship with the Ministry. While private sector engagement with the Ministry is a valid goal—this engagement is about advocacy and perhaps planning. The evaluation team did not find any interest on the part of the private sector partner to coordinate specific interventions in improving literacy directly with the Ministry. It is expecting this role to be filled by AMCHAM/DR. Brand interventions. In addition, BEP’s outsourcing of its technical assistance should be examined more closely by an internal review and BEP should work with vendors to brand BEP. To assure sustainability of the private sector model and BEP’s program, directors and teachers must know that BEP and the sponsor are working in the school. Focus intervention. BEP does not have sufficient staff resources to perform against the targets included in the cooperative agreement and address all of the needs that the beneficiary schools have. Coordination will help leverage BEP resources. However, focus is also necessary. The evaluation team recommends that USAID, AMCHAM/DR, the private sector sponsors, and MINERD discuss (collectively or bilaterally) a 8 This cannot be measured at this time due to the lack of data. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 15 niche for BEP. Based on the classroom observations, interviews with directors, and discussions with teachers, there are several potential areas of focus that may vary across schools: • Early grade reading • Classroom management and discipline • School-related violence The evaluation team does not believe it should provide a recommendation for the area of focus because of the need for stakeholder consensus and mapping of actors at the school level. Based on the data collected, it is plausible that some other entity or mechanism might be the best means to meet some of the needs. The conclusion and recommendation is that attempting to solve all of the needs with the resources available to BEP will continue to limit its effectiveness. A niche that is integrated into the overall framework and customized to the school is a more appropriate and achievable goal. Measure Outcomes. BEP has underway an analysis of several of the interventions that it has and will implement. This analysis should include a cross-cutting examination of the results among sponsors and schools. A results summit should be held at the school-level, with the sponsors, with USAID, and perhaps among USAID projects (see Recommendation #3). Recommendation #2: BEP should redesign its overall organizational and monitoring structure for its operations to be school-focused rather than vendor-focused. To address the following conclusions: Conclusion #2: BEPs written records are insufficient for effective monitoring of the M&E Plan or to achieve the Result No. 1 under Component No. 4. Conclusion #9: BEP’s management organization for its files obfuscates what is happening at the school-level. Key informant interviews revealed that sustainability for school sponsorship depends on sponsors seeing that the community recognizes their good will as well as improvement at the school-level. Based on the key informant interviews with sponsors, AMCHAM/DR’s role is to facilitate the delivery of technical assistance that improves the quality of education in public schools. The AMCHAM/DR value proposition for the private sector partners is the comprehensive, coordinated and systematic interventions that it manages and then monitors for performance. To realize this key purpose, AMCHAM/DR must have adequate records at the school-level. By organizing its records by contract or agreement, AMCHAM/DR is diverting its focus. While raising funds is an important function, it is just that—a function and not a purpose. The following outlines the key elements of this management organizational structure that becomes school-focused: • Restructuring of the data collection plan to be school and director/teacher based. A simple, basic information sheet on the school should be developed that records all assistance provided, all phone calls, and all monitoring visits for each academic year. • Instituting a standardized reporting requirement for all vendors that at a minimum includes: date of assistance, names of persons receiving training or technical assistance, sex, years of experience, associated school and school code, years at school, intensity of assistance, grades of instruction, and number of students in class. • Developing a standard type of reporting mechanism and procedure with the schools. It is insufficient to include only a requirement for reporting in the MOU as it is unlikely the data made available will be sufficient. • Mapping of the institutions operating at each school at the beginning and end of each school year. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 16 • Expansion of site visits to include the BEP Director, Education Coordinator, and the new staff member.9 • Reformulation of the annual report to USAID to be school-based with a review of achievements by school. • Regular meetings to be held with MINERD. • Regular joint meetings with sponsors that includes a school-sponsorship learning summit. Recommendation #3: Lessons learned from this evaluation related to ESP should be examined broadly and include BEP and the sponsors. Recommendation #4: BEP should include in its assessment of private sector sponsorship a close examination of the role of strategic alliances. To address the following conclusions: Conclusion #5: Private sector involvement as a sponsor and working through AMCHAM is on track to be sustainable. Conclusion #6: Without a clear strategy for building a strategic alliance for intervention at the school-level, BEP may sustain interventions without achieving improvements in education. Conclusion #7: Access to materials (equipment and didactic tools) and training of teachers are necessary but not sufficient condition to change behavior. The purpose of this mid-term evaluation excluded a focus on why private sector partners decide to become school sponsors or how to expand school sponsorship. This is an objective of the upcoming assessment. However, as a means to verify if the perceptions of key stakeholders match the observations at the school level, the evaluation team interviewed sponsors. A key conclusion is that AMCHAM/DR and BEP offer a sustainable means to channel private sector involvement at the school-level. The constraints for success include the capacity for BEP to articulate a basic education strategy and coordinate at the school-level. Although the donation of materials is an efficient means to channel private school sponsorship, all stakeholders recognize it is insufficient. Thus, the evaluation team recommends that USAID organize an experience or lessons learned summit with its education portfolio that includes sponsors and identifies lessons learned in terms of the mechanisms for delivery and coordination. In addition, the upcoming private sector sponsorship assessment should include specific questions about strategic alliances with MINERD, AMCHAM/DR, the private sector sponsors, the other three entities focused on basic education, and the school. USAID can facilitate the dialogue with PUCMM. Strategic alliances with OEI and Poveda are likewise important. 9 While there is a plan to add a staff member who is responsible for monitoring and evaluation, this new position should be an education specialist. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 17 Appendices Appendix C-1: Descriptive Data about BEP Interventions Upon request the evaluation team can provide the excel files for the following tables. Table 1: Number of Schools (total =51) that received the intervention Sources: (PRISMA Febrero 2012); (PRISMA Enero 2011); (PRISMA Abril 2012); (Basic Education Program March 2012); (EDUCA 2012); (EDUCA 2012); BEP files. Table 2: Distribution of Schools by Geographic Area Source: (Basic Education Program March 2012) By Intervention Component Totals (Schools) % 1A PRISMA's five workshops (includes five workshops prior to current cooperative agreement) 35 69% 1B Directors Training August 2012 38 75% 1C Animación en Lectura August 2012 (no data available) 0 2 Donation of materials 44 86% 3 EpCs for third and fourth graders (updated based on data available); Data for Constanza schools not disaggregated by school (only community) 19 37% 3.1 School Year 2009-2010 9 3.2 School Year 2010-2011 15 3.3 School Year 2011-2012 19 4 Quantum Learning for teachers (ENTRENA) [number reported by BEP, no data available to verify]. 23 45% 5 ESP with PUCMM (per BEP, but no independent verification, contract with PUCMM cites 2 schools) 1 2% 6 Other Interventions 2 4% By Geographic Location No. of Schools Barahona 5 Bayahibe 1 Boca Chica 1 Cevicos 1 Constanza 10 Hacienda Estrella 2 Haina 1 Haina - San Cristobal 1 Higüey 1 La Barquita Santo Domingo 1 La Vega 1 Najayo 1 Najayo, San Cristobal 4 Pedernales 1 San Cristobal 2 San Pedro de M. 1 Santiago 2 Santo Domingo Este 4 Sto Dgo Oeste, Palabe. 1 Villa Altagracia 6 Villa Altagracia - Batey lecheria 1 Villa Mella 3 51 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 18 Table 3: Distribution of Schools by Sponsor (as of March 2012) By Private Sector Partner No. of Schools FUNDACION RICA 25 49% INCA 2 4% Fundación Futuro posible 1 2% CEPM 2 4% Ege-Haina 3 6% AES Dominicana 4 8% San Diego Padres 4 8% IMCA 3 6% Fundación Propagas 2 4% Synergies 1 2% Maritima Dominicana 1 2% PISSA 1 2% Comite de Santiago 1 2% Comite de La Vega 1 2% 51 Table 4: Available Data on Teachers and Directors Trained by Ediciones SM Sources: (PRISMA Febrero 2012); (PRISMA Enero 2011); (PRISMA Abril 2012); By Private Sector Partner No. of Schools 2009 Teachers Trained 2010 Teachers/ Directors Trained 2011 Teachers/ Directors Trained 2012 Teachers Trained 2012 Directors Trained FUNDACION RICA 25 106 16 64 0 20 INCA 2 0 0 27 16 1 Fundacion Futuro posible 1 0 0 0 0 1 CEPM 2 0 0 0 0 1 Ege-Haina 3 0 0 0 0 2 AES Dominicana 4 0 38 0 0 4 San Diego Padres 4 18 0 0 0 4 IMCA 3 0 0 0 0 3 Fundación Propagas 2 0 0 0 0 2 Synergies 1 0 0 0 0 0 Maritima Dominicana 1 0 0 0 0 2 PISSA 1 0 0 0 0 1 Comite de Santiago 1 0 0 0 16 0 Comite de La Vega 1 0 0 0 20 1 14 51 124 54 91 52 42 Notes: These are the data as reported by Ediciones SM and does not include any training after July 2012 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 19 Appendix C-2: Schools Visit and Classroom Observations Table 5: Schools Visited by Evaluation Team and Interviews Conducted Tables 6: Distribution of Classrooms Observed REGION # % SANTO DOMINGO 6 40% COTUI 2 13% LA VEGA 2 13% SANTIAGO 0 0% PUERTO PLATA 0 0% SAN CRISTOBAL 5 33% TOTAL 15 ZONA # % URBAN 11 73% RURAL 4 27% TOTAL 15 GRADE # % 1 0 0% 2 4 27% 3 4 27% 4 5 33% MULTIGRADE 1&2 2 13% MISSING 0 TOTAL 15 SUBJECT OBSERVED # % LE 6 40% MAT 4 27% OTHER 5 33% TOTAL 15 Tables 7: Classroom Size (Attendance) and Teacher Years of Experience STUDENT TEACHER EXPERIENCE (Years) Mean 29.26667 Mean 15.75 Standard Error 2.287579 Standard Error 2.421886 Median 29 Median 16.5 TEACHER'S EDUCATION # % LIC. BACH 5 42% MASTER 0 0% OTHER 7 58% MISSING 3 TOTAL 15 Date School BEP # Code Selected Grouping Actual Grouping Period Director interview Teacher interview(s) Classroom Observation(s) 10/11/2012 Villa Pinales 7 01952 4.00 5.00 Afternoon 1 1 1 10/16/2012 Carmen Celia Balaguer 11 00196 4.00 5.00 Afternoon 1 2 0 10/16/2012 Escuela Narciso Alberto 14 03728 2.00 3.00 Afternoon 1 2 2 10/16/2012 Elias Polanco 24 03728 0.00 1.00 Morning 1 1 2 10/9/2012 Francisco del Rosario Sanchez 27 00350 2.00 3.00 Morning 1 3 4 10/11/2012 Pueblo Nuevo 34 00081 2.00 3.00 Morning 1 1 1 10/9/2012 Bajos de Haina 37 03217 4.00 4.00 Afternoon 2 1 2 10/11/2012 La Playa 40 03034 1.00 2.00 Afternoon 1 1 1 10/11/2012 San Martin de Porres 45 05395 2.00 3.00 Morning 1 5 1 10/11/2012 Colonia Japonesa 46 01940 3.00 4.00 Morning 1 1 1 11 18 15 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 20 Tables 8: Classroom Observations: Adoption of Learning Strategies EDUCATION MATERIALS ON WALLS # % COVERED W/ RELAVENT MATERIALS 8 53% SOME RELAVENT MATERIALS 4 27% LITTLE TO NO MATERIALS 3 20% TOTAL 15 LEARNING CORNERS # % HAVE 10 67% DO NOT HAVE 5 33% MISSING 0 TOTAL 15 CLASSROOM ORGANIZATION # % TEACHER AT FRONT, STUDENTS IN SEATS 12 80% STUDENTS IN PAIRS 0 0% STUDENTS IN SMALL GROUPS 3 20% NOT WORKING 0 0% MISSING 0 TOTAL 15 GENDER BALANCE # % GIRLS AND BOYS EQUAL OPPORTUNITIES 7 47% GIRLS GIVEN MORE OPPORTUNITIES 1 7% BOYS GIVEN MORE OPPORTUNITIES 5 33% LITTLE OR NO PARTICIPATION 2 13% TOTAL 15 Figure 1: Classroom Observations (n=15) Q13A1= WRITES ON BOARD FOR DISCUSSION Q13A2= WRITES ON BOARD TO ANSWER QUESTIONS Q13A3= WRITES ON BOARD TO HAVE STUDENTS COPY Q13A3= WRITES ON BOARD TO SUMMARIZE Q13A4= WRITES ON BOARD TO HAVE STUDENTS READ Q13B1= TEACHER DICTATES Q13B2=TEACHER READS ALOUD Q13B3= TEACHER ASKS STUDENTS QS INDV Q13B4= TEACHER ASKS STUDENTS QS GROUP Q13C= TEACHER DEMONSTRTES PHYSICALLY Q13D= TEACHER DEMONSTRATES WITH MATERIALS Q13E= TEACHER DRAWS Q13F= TEACHER USES POSTER Q13G= STUDENTS READ ALOUD Q13H= STUDENTS READ SILENTLY Q13I= STUDENTS ROLE PLAY Q13K= STUDENTS DEMONSTRATE PHYSICALLY Q13L= STUDENTS DEMONSTRATE WITH MATERIALS Q13M= STUDENTS DO SEATWORK Q13M= OTHER 0% 20% 40% 60% 80% 100% Q13A1 Q13A2 Q13A3 Q13A4 Q13A5 Q13B1 Q13B2 Q13B3 Q13B4 Q13C Q13D Q13E Q13F Q13G1 Q13G2 Q13H Q13I Q13K Q13L Q13M Q13N Advanced techniques, low adoption Participatory techniques utilized at high rates of adoption AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 21 Appendix C-3: EpC Programmatic Data Table 9: Children Served in EpCs (2011-2012) by EpC Name, School, and Sponsor Source: EDUCA Database for EpC Names and EDUCA Report to BEP for School Names. It is not possible to match exactly and this information is not available to BEP. The above is a draft attempt to align the data. Table 10: Children Served in EpCs (Period Unknown) Source: EDUCA database. Espacios School Name Sponsor Children Sponsored Reported BARAHONA, VILLA CENTRAL María Montes 33 30 BASIMA VILLA ALTAGRACIA Felicia Cuesta Fundación Rica 58 62 CACHON BARAHONA Fidel Medina Fundación Rica 85 60 COLONIA JAPONESA CONSTANZA Colonia Japonesa Propagas 38 30 COLONIA KENNEDY CONSTANZA Colonia Kennedy Fundación Rica 31 EL TANQUE BARAHONA Albida Maria Santana Ege Haina 37 85 ENSANCHE LIBERTAD CONSTANZA Community Fundación Rica 30 ENSANCHE LIBERTAD CONSTANZA Community Fundación Rica 30 HACIENDA ESTRELLA ESTRELLA VILLA MELLA Hacienda Estrella Fundación Rica 84 30 KM 17, BAJOS DE HAINA Bajos de Haina AES Dominicana 41 60 KM59 VILLA ALTAGRACIA Km 59 Fundación Rica 26 24 LA SABINA CONSTANZA Community Fundación Rica 30 MIRADOR DUARTE CONTANZA Community Fundación Rica 29 PALERO ABAJO CONSTANZA Community Fundación Rica 29 PESCADERIA BARAHONA Luis Felipe Feliz Fundación Rica 54 30 SAN FELIPE VILLE MELLA San Felipe Fundación Rica 29 31 SOLARES DE MILTON BARAHONA 31 VILLA PINALES CONTANZA Villa Pinales Fundación Rica 30 CONSTANZA COMMUNITIES 210 Total 725 652 * (chart had total of 627 60 712 (per amount reported) U de t ed C sc oo a e a, a d u dac ó ca Sc oo ac e da st e a (Ca e Ce a Balaguer) and Barahona school (María Montes is a beneficiary school, but not clear if Villa Central or Solares del Milton). 33 58 85 38 31 37 30 30 84 41 26 30 29 29 54 29 31 30 0 10 20 30 40 50 60 70 80 90 NNA Atendido AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 22 Table 11: Percentage of Girls and Boys Served in EpCs (Period Unknown) Table 12: Reason for Participation: Opportunity or Overage 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Fem Masc 0 10 20 30 40 50 60 70 80 90 sobre edad oportuna AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 23 Table 13: Percentage Distribution of Problem Area for Students by EpC Table 14: Data Available on Status at End of School Year (n=725) 0 10 20 30 40 50 60 70 80 90 BARAHONA, VILLA CENTRAL BASIMA VILLA ALTAGRACIA CACHON BARAHONA COLONIA JAPONESA CONSTANZA COLONIA KENNEDY CONSTANZA EL TANQUE BARAHONA ENSANCHE LIBERTAD CONSTANZA ENSANCHE LIBERTAD CONSTANZA HACIENDA ESTRELLA ESTRELLA VILLA MELLA KM 17, BAJOS DE HAINA KM59 VILLA ALTAGRACIA LA SABINA CONSTANZA MIRADOR DUARTE CONTANZA PALERO ABAJO CONSTANZA PESCADERIA BARAHONA SAN FELIPE VILLE MELLA SOLARES DE MILTON BARAHONA VILLA PINALES CONTANZA LE and Math LE Mat No Data 0 50 100 150 200 250 300 350 400 450 500 LE and Math LE Mat Repeat No Data Deficiency at start of EpC Levels that passed at end of the year AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex C: Basic Education Program 24 Appendix C-4: Documents Reviewed In addition to the work cited, the evaluation team reviewed all quarterly and annual reports and the vendor reports made available by AMCHAM/DR. Bibliography Basic Education Program. "First Quarterly Report: Oct 2010-Jan 2011." 2011. Basic Education Program. "General Proposal for Basic Education Program." 2012. Basic Education Program. Interventions List. Santo Domingo: AMCHAMDR, March 2012. Basic Education Program. "Lineamientos para el monitoreo y seguimiento del proyecto." 2012? Basic Education Program. "Project Monitoring Report for 3rd Quarterly Report: Apr-Jun 2012." 2012. Basic Education Program. "Project Monitoring Report-2nd Quarterly Report for Jan-March 2012." Jan￾March 2012. —. "Propuesta Intervención escuelas 2012 to Fundación Rica." December 23, Dec 2011. —. "Propuesta Intervención escuelas, año 2012-2013 to Asociación Popular de Ahorros y Préstamos." June 18, June 2012. Basic Education Program. "Quarterly Report: Feb 2011-May 2011." 2011. Basic Education Program. "Quarterly Report: Oct 2011-Dec 2011." 2011. Basic Education Program. "Second Quarterly Report: February - March 2010." 2010. Basic Education Program. "Summary of Activities: October 2009-June 2010." 2010. Basic Education Program. "Workplan October 2010 - September 2011." 2011. EDUCA. Database for 2011-2012 Year. Santo Domingo, 2012. EDUCA. "Reporting Tables for EpCs 2009-2012." 2012. EDUCA. "Resumen de Capacitaciones Facilitadoras EpC 2009-2012." Santo Domingo, 2012. Entrena. "Proposal." Santo Domingo, February 2012. PRISMA. "Escuela de directores: Gestión educativa basada en resultados." Santo Domingo, Febrero 2012. PRISMA. "Informe Técnico Ciclo de Capacitación 1: Sept-Dic 2010." Santo Domingo, Enero 2011. PRISMA. "Informe Técnico-Segundo Ciclo de Capacitación 1: Sept-Dic 2011 y Enero-Marzo 2012." Santo Domingo, Abril 2012. USAID. "Attachment B Program Description for Basic Education Program." Dominican Republic, 2009. USAID Contract with AMEX International. "Intergrated Performance Evaluation of USAID/Dominican Republic Education Portfolio." 2012. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY i Annex D: Out of School and At Risk Youth and Children Programs (ARCY) Submitted to: USAID/Dominican Republic Prepared by: John Helwig, Ph.d., Evaluation Specialist, ARCY Edited by: Virginia Lambert, Team Leader for Portfolio Evaluation January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 18 USAID/DR Out of School and At Risk Youth and Children Programs (ARCY) Annex D: Table of Contents 1. Introduction ............................................................................................................................................. 1 2. Evaluation Methodology......................................................................................................................... 1 3. Findings, Conclusions, and Recommendations.................................................................................. 1 3.1 DREAM Grant.................................................................................................................................... 1 Performance............................................................................................................................................. 2 Efficiency .................................................................................................................................................. 2 Sustainability ............................................................................................................................................ 3 Recommendation(s) ................................................................................................................................. 3 3.2. CRS Grant......................................................................................................................................... 4 Performance............................................................................................................................................. 5 Efficiency .................................................................................................................................................. 6 Sustainability ............................................................................................................................................ 6 3.3 IDDI Grant.......................................................................................................................................... 7 Performance............................................................................................................................................. 8 Efficiency .................................................................................................................................................. 8 Sustainability ............................................................................................................................................ 9 Recommendation(s)................................................................................................................................. 9 3.4 Counterpart International ..............................................................................................................10 Performance...........................................................................................................................................10 Efficiency ................................................................................................................................................11 Sustainability ..........................................................................................................................................11 Recommendation(s)...............................................................................................................................11 3.5 Alfalit Grant .....................................................................................................................................11 Performance...........................................................................................................................................11 Efficiency ................................................................................................................................................12 Sustainability ..........................................................................................................................................12 3.6 Major League Baseball-Dominican Development Alliance ........................................................12 Performance...........................................................................................................................................13 Efficiency ................................................................................................................................................13 Sustainability ..........................................................................................................................................14 Recommendation(s)...............................................................................................................................14 3.7 Batey Community Development Project ......................................................................................15 Performance...........................................................................................................................................15 Efficiency ................................................................................................................................................16 Sustainability ..........................................................................................................................................17 Recommendation ...................................................................................................................................17 Appendix D-1: Bibliography.....................................................................................................................17 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 1 1. Introduction In 2008/2009 USAID/DR made five small grants to support programs that focus on out-of-school and at￾risk youth. This report provides a review of the accomplishments and lessons learned from these small grants. This report also examines the Major League Baseball Dominican Development Alliance (MLB￾DDA) and provides a mid-point review of the education components of the Batey Community Development (BCD) project. The evaluation team that conducted the review of the ARCY programs simultaneously conducted a mid￾term evaluation of two of USAID/DR’s biggest education programs (BEP and PEF), as well as a portfolio review of all education programs implemented by USAID/DR. This report is an annex of the Education Portfolio Evaluation Integrated Report (Integrated Report) and contributes to the Integrated Report to the extent that lessons learned may be relevant to education programming and may influence decisions about future programming. Lessons learned can also be used to inform an At-Risk Youth Program that started in 2012. This review examines each of the ARCY grants in three key areas: performance, sustainability, and efficiency. Below the methodology used to review the grants is outlined. 2. Evaluation Methodology For a detailed description of the methodology employed in the USAID/DR Education Portfolio Evaluation see the full Integrated Report. Data specific to this individual report includes review of documents provided by USAID and implementing partners, interviews with project managers and beneficiaries, as well as visits to project sites. 3. Findings, Conclusions, and Recommendations This section of the report examines each of the 5 small grants as well as the MLB-DDA and the education components of the BCD project. This section is divided into subsections based on each grant. Within these subsections a brief overview is provided on the grant program followed by a summary of information regarding grant performance, efficiency, and sustainability. (More information on the definitions of how the evaluation team evaluated each of these areas is provided in the integrated report.) Several brief recommendations are provided for each grant. It is worth noting that in most cases the small grants programs had completely ended. Therefore, information on efficiency is limited, because of the difficulty of collecting information on the efficiency of project management and implementation after a project has closed. 3.1 DREAM Grant Box 1: Dream Grant At a Glance Partner Program Summary DREAM – Dominican Republic Education and Mentoring Project $298,500 2/1/08 – 9/30/09 Cabarete Two four-week summer school and camp programs to serve 250 youth/adolescents in 2008 and 250 in 2009. Objectives Objective: Devote resources to developing and implementing quality academic tutoring programs that focus on literacy, writing, math and ESL that target at-risk youth, defined as those who are more than two grades behind their age group, those who are not attending school, those of Haitian heritage and those who have learning and physical disabilities. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 2 The DREAM program is implemented through two summer camps, which were held on time at the two proposed sites in Cabarete. The two four-week summer programs ran from July 7th- July 31st, 2008, with 252 participants and from July 6th to 30th in 2009, with 271 participants. The Youth Camp for ages 10-12 focused on math, reading, writing and life skills using art, music, dance, sports, swimming, science and nature. Participants were also taken on field trips to an amusement park, a museum and nature park. The Adolescent Camp for ages 13-18 focused on developing employable skills. Reading, math, health and English were offered along with sports and computer classes. Five mini￾courses included preparing resumes, basic job skills, writing a newsletter, and an art group. In the first year DREAM employed 101 and in the second year 111 volunteers, staff, and support staff of which 20% were Dominicans. Ninety three percent of students in the 2009 program were at-risk. In the 2009 Youth program, 34% of the students were two grade levels or more behind in school or were not attending school and 11% of the students were of Haitian heritage. In the 2009 Adolescent Program nearly 60% were not enrolled in school or were two grade levels or more behind in school and 9% of enrolled students were of Haitian heritage. In addition, the Adolescent Program served one severely disabled student with muscular dystrophy, whose wheelchair has prevented him from attending public school, making DREAM summer and preschool programs his only exposure to education. Performance In both years, the ratio of 55% female and 45% male met the goal of enrolling more females by a margin of ten percent. A total of 523 youth and adolescents benefited from the program. In 2008 a reading assessment, using the Rigby Running Records System, was conducted in the Youth Camp showing a 2.6 points growth, and in the Adolescent Camp there was a growth of 3.94 points. This assessment tool was developed by Houghton, Mifflin, Harcourt and reading materials and assessment tools are available in Spanish.1 The project kept on track, surpassed the targets (targets were 250 youth per summer) and is still on track without USAID financial support. The evaluation team visited several classrooms where volunteers were working with small groups of children. In one room a volunteer was using an innovative strategy that caught the attention of the team: the “Letter of the Day”. If, for example, the letter of the day was “B” children were asked to brainstorm and make a list of as many words as they could, in a given period of time that began with “B”, and then use the words in a sentence. In this manner the teacher could test whether they comprehended the words. In other rooms children were drawing or painting, and in others, quietly reading or being read to by a volunteer. Efficiency The fact that the summer program has continued to be implemented in the three years following the completion of this grant and has also resulted in a spin-off program called Young Stars, that provides services for youth during the school year is evidence that the program(s) are efficient. Young Stars is focuses on personal growth, social and academic skill development. 1 See: rigby.hmhco.com. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 3 The design and management of the program(s) is efficient. DREAM was able to maintain a 4:1 student/teacher ratio at all times. This was achieved through using volunteers and staff requiring only a small incentive stipend. However, in the future it would behoove DREAM to train and employ a greater number of Dominican staff. Sustainability DREAM has been able to engender enough donor support from local businesses and expat residents who vacation in Cabarete to continue the summer program in Cabarete and implement the Young Stars program that provides continuing support to youth throughout the school year. DREAM seems to be well accepted in Cabarete and is able to raise funds locally to support, continue and expand their work in this community and environs. There continues to be demand for DREAM’s programs; the Summer Camp program has continued for three years following the termination of USAID support. Much of the appeal is due to the presence, interest and commitment of the volunteer expats. The evaluation team was told that parents believe that this interaction between their children and the expats enriches the lives of their children. Two members of the evaluation team visited DREAM in Cabarete and met with the Executive Director and also greeted many of the staff who are working with children in the Montessori and Young Stars programs. It appeared that all of the managing staff are ex-pats, many of whom are transient. There are some Dominican teachers and support staff, but the technical as well as administrative supervision and management appears to be undertaken by the ex-pats. With a primarily ex-pat and transient managerial and technical staff, and funding primarily from expats and international sources, the evaluation team would hesitate to recommend that DREAM is able to or should expand their work outside the area where they presently work, or at least until they are able to hire permanent, qualified Dominican staff and obtain more funding from Dominican sources. Key informant interviews suggest that MINERD officials are hesitant about working with institutions that come to them asking for support for the institutions’ programs. Officials prefer to first discuss needs, interests and problems, and later be involved in developing programs that respond to what they perceive as needed and what can be accomplished. Recommendation(s) 1. Dialogue with MINERD should be frequent and involve all parties as equal. This coordination must continue throughout programs to ensure that efforts are not duplicated and that funds (both from the GODR and the USG) are efficiently spent. 2. DREAM has demonstrated significant prospect for sustainability. Given this, the program should continue to concentrate its programs within the immediate area of Cabarete where it enjoys high respect and cooperation from the community. If, in the future, the management team includes a higher percentage of qualified Dominicans, and is able to find private donors and engender coordination with public entities in new areas (e.g. Sosua, Puerta Plata), DREAM might be able to extend its area of service. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 4 3.2. CRS Grant Box 2: CRS Grant At-a-Glance Partner Program Summary CRS – Catholic Relief Services $610,500 5/20/08 – 5/19/10 The project received a one-year extension to May 2011 and USAID provided an additional US$100,000. Santiago, Boca Chica and Santo Domingo Serve 950 children and youth in: 1. Providing Math and reading skills to bring up grade level for retention and/or reinsertion. 2. Providing support services (health, legal, shelter, psychological, conflict and cultural activities). 3. Working with communities (leaders, families, schools) to provide support and monitoring. 4. Building capacity for 5 local partners. Objectives 1: Boys, Girls and Adolescents (BGA) have access to alternative educational opportunities to acquire math and language learning skills, as well as other practical skills to bring them to the right grade level for their reinsertion and retention in school. 2: For OOSARCY to focus on the physical, psychological, social and learning environment to ensure the development of necessary skills to participate in school, family, and the community as productive members of society. 3: To create and maintain a synergy of support for the education of the BGAs. The CRS grant identified 2,258 at-risk or out-of-school children and adolescents of whom 934 participated in 3 or more project services and completed project programs. 898 participated in an alternative education program. 184 children returned to school during the LOP. Computers for LE learning were installed in the 5 alternative education centers established by the project. 128 adolescents received vocational training certificates. 220 parents/ caretakers and 30 community leaders participated in project programs for the BGAs. The project created spaces for learning and a program designed so that the participants could define a Plan for Life with a clear idea of how to achieve a productive and satisfactory adulthood. They endeavored to create an environment where they could offer guidance, recreational opportunities and protection, while developing stronger families and communities. Through the implementation of the “Learning Together II” Project, the NINA Consortium has strengthened and intensified the project components for: a) integrating and keeping BGAs in schools; b) the commitment and training of families and leaders; c) creation of a support and referral network between teachers, schools and Alternative Education Centers (AECs) to improve student retention, re‐insertion; and d) implementing the alternative and family curriculum. The four components were implemented by Acción Callejera, Caminante Proyecto Educativo, Programa Quédate con Nosotros of Muchachos y Muchachas con Don Bosco, Niños del Camino and Programa Yo También of the Catholic Youth Pastorate, and led by CRS, in 6 communities within Santiago, Boca Chica, Santo Domingo Norte and the Central District of Santo Domingo. CRS coordinated the design, development and implementation of programs with each of the consortium members. Each consortium member organized and implemented the alternative education program and provided psychological, legal, medical and other services to families according to each members ability. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 5 Performance Program participants are attending school and in most cases achieving at higher rates, their comportment is improved and they demonstrate better personal and social skills. Parents/caretakers are better prepared and disposed to support their BGAs in their schooling. Table 1: Achievement of targets for Strategic Objective 1 INDICATOR FISCAL YEAR 1 FISCAL YEAR 2 FISCAL YEAR 3 May – Sept 2008 Oct 2008 – Sept TOTALS 2009 Oct 2009 – May 2010 PLANED ACHIEVED PLANNED ACHIEVED PLANNED ACHIEVED PLANNED ACHIEVED 1. 713 at-risk BGA improve math skills 80 167 404 521 229 210 713 898 2. 713 at-risk BGA improve language skills 80 167 404 521 229 210 713 898 The chart shows that in relation to the original goal, a larger number of BGAs received remedial education. The final goal was accomplished with 898 instead of the 713 that were originally planned. At the end of the project, 229 BGAs were tested; and 185 who had already concluded the academic leveling program, improved their language and math skills. Table 2: Achievements of Result 1.1 INDICATOR FISCAL YEAR 1 FISCAL YEAR 2 FISCAL YEAR 2 May – Sept 2008 Oct 2008 – Sept TOTALS 2009 Oct 2009-May 2010 PLANNED ACHIEVED PLANNED ACHIEVED PLANNED ACHIEVED PLANNED ACHIEVED 200 BGA mainstreamed to school 75 85 105 89 20 7 200 184 750 at-risk BGA kept in school and/or AEC 130 532 500 956 120 220 750 1708 It was possible to mainstream into school 181 BGAs; and it was possible to keep in school 1,708 who had been identified as being at risk of dropping out of school. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 6 Two of the five consortia members were able to achieve certification for managing funds by international donors. However, the other three will continue to require assistance and operate under the supervision and budgetary control of certified organizations. Efficiency The design and overall management of the project was efficient. Efficiency, specifically in the area of M&E could have been improved. The final evaluation reports that due to the inefficiency in data recording, collection and reporting, results probably were better than those shown. Sustainability These programs can only be implemented if funding is available. Members of this consortia are not self￾supporting, although they all have some degree of relationship with the Catholic Church and receive some regular funding from both The Church as well as from CRS International. As long as funds are available, the consortia members can employ or attract volunteers-with-incentives to provide other social protection services (medical, legal, psychological, etc.) for at-risk families. CRS endeavored to strengthen the capacity of its five associate local NGOs who were responsible for implementing the program so that they would be able to manage grants on their own. These NGOs included Caminante Proyecto Educativo in Boca Chica, Acción Callejera (AC) in Santiago, and Quedáte Con Nosotros, Yo También Pastoral Juvenil and Niños del Camino in greater Santo Domingo. Caminantes and AC demonstrated sufficient capacity to manage funds. It is somewhat likely that Caminantes and Acción Callejera will continue providing math and reading support to bring students up to grade level, but it will depend on funding and the continued training and capability of personnel. Caminantes is currently implementing alternative education programs under contract to ENTRENA with funds from ACNUR. It is somewhat unlikely that other consortia members will continue providing these academic services. Consortium member organizations are well embedded in their communities and are equipped to provide support services (health, legal, shelter, psychological, conflict and cultural activities) as long as they have financial support. The project experienced problems working with local and national authorities, although they were able to build synergy between families and schools. When the project was granted a one-year extension UNICEF also provided counterpart US$85,000 and donated 5 computers and 5 projectors. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 7 3.3 IDDI Grant Partner Program Summary IDDI – Institituto Dominicano de Desarrollo Integral $199,000 5/20/08-12/30/10 Cotui The goal of the School Reinsertion and Promotion of Productive Development Program for Youth project was to have 400 youth generating income while creating the platform to advance grade levels by core learning (leadership, life/work skills; technical/vocational training and micro-enterprise). Objectives General Objective: The School Reinsertion and Promotion of Productive Development Program for Youth 13-21, Residents of the Municipality of Cotuí, Sanchez Ramirez Province was conceived to take advantage of the corporate responsibility of the Barrick Gold Corporation to support the economic and social development of the Province, specifically through the (re)integration of vulnerable youth to academic and productive environments. The project aimed to have USAID, the Barrick Gold Corporation with an equal contribution and IDDI with a 5% contribution finance the program. IDDI as lead managed the program, together with sub-grantees ENTRENA and ITECO. Specific objectives: • Participating youth acquire life and vocational skills as well as micro-enterprise knowledge. • Participating youth acquire abilities that allow the insertion into the labor market. • A framework for project sustainability and replication is created. • Gender equity and knowledge on reproductive health on the beneficiary population favorably improves. The project was planned and all components implemented over a period of two years with the participation of IDDI, ENTRENA, ITECO and USAID, and was implemented in 15 communities in Sánchez Ramírez Province. In general terms, these communities are characterized by the marked poverty of their inhabitants, few job opportunities, limited access to basic services and informal economic activity. Learning centers called Spaces for Entrepreneurship (EpEs), with 20 to 25 youth each, were established in locations provided by churches and other civic groups. The EpE consisted of three components: a) Core-Learning to learn, Quantum leadership and Live/Work Skills (interviewing, marketing yourself, resumes), b) Technical/Vocational training (basic English, computer, specific technical) and c) Micro-enterprise. Some centers offered the complete EpE program and others only the microenterprise component. The EpE program is based on the opportunity that the young people will have to participate in a program that will equip them with the strategies required to be able to find a job or establish a micro-enterprise or continue formal education or both work and study. The program provides youth participants with an opportunity to begin again and demonstrate that he or she can continue to learn. The program is also based on equality, since, often these youth have not had the same opportunities as other youth, and should be provided the same chance that others have been provided. Barrick Gold Corp. backed out of the program after having agreed to provide matching funds plus on-job training, transportation and food for trainees. Barrick eventually contributed a sum equivalent to about AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 8 one-fifth of what they had committed to providing originally, and this was used to purchase a van to transport participants and staff to and from rural towns and villages. In spite of this setback, and the reduced budget, IDDI/ENTRENA were able to provide microenterprise and full EpE training for 252 beneficiaries. IDDI was able to recruit and train beneficiaries thanks to alliances and agreements made with local organizations, such as training centers, churches, INFOTEP, neighborhood associations, among others. Performance A total of 307 youth were enrolled in the EpE program, and 252, or 82%, completed the program. These 252 beneficiaries completed a total of 450 hours, in two periods, attending 3 hours per day, 5 days a week, for 8 months in each period. Seventy five of the beneficiaries had previously abandoned school, and by the end of the project 49, (32 males and 17 females) had returned to school. In terms of the program’s specific objectives (see above), the following achievements are noted: Participating youth acquire life and vocational skills, and knowledge on micro-enterprises. IDDI did manage to enroll 307 beneficiaries in the EpE program of which 82% or 252 completed the program, and 65% of those who had abandoned schooling had returned. Participating youth gain skills through vocational training that will allow them to enter the labor market. 85% of participants were trained in technical skills that prepare them to manage their own micro￾enterprise or obtain employment. Favorably improve gender equity and reproductive health knowledge of the target population. 23 young mothers trained in different technical skills; 4 women have started a micro-enterprise, 2 of which are young mothers; 3 women have secured employment (one of them is a young mother); 17 women are re￾inserted in the formal education system; 51 youths were trained to share information about healthy lifestyles, sexual and reproductive health and gender equity; 4,602 youth were reached through conferences and workshops, out of which 2,891 were females and 1,711 males. The evaluation team interviewed 8 youth and 2 facilitators who had completed the program two years ago. All have continued their education and also were earning income and were actively pursuing goals set while they were participants in the program, and continued to meet with and support each other. Program managers who accompanied the evaluation team reported that the self-esteem of participating youth significantly improved with the implementation of the project. The young people expressed a hopeful vision of the future and a desire for personal improvement and empowerment. Despite the setback caused by Barrick Gold Corp.’s withdrawal of support, the project was able to keep on track, although it was unable to meet it’s original target of 400 youth. Efficiency The execution of the program encountered problems when Barrick backed out, however, IDDI/ENTRENA were able to find other local partners who provided some basic goods and services that enabled the project to efficiently fulfill its mandate, although with a reduced beneficiaries. Public-Private partnerships are very difficult to arrange and even more difficult to maintain. There is no easy answer or solution. In this case, Barrick had committed in writing to providing a matching sum to that of USAID, but it was not AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 9 feasible to hold this Canadian-based firm to its commitment as legal action would take a long time and the project had to be completed within USAID’s timeline. Sustainability A framework for project sustainability and replication was created. The project trained a group of instructors and facilitators who remain in the Province. Unless funding is provided it is highly unlikely that the program can continue and their knowhow will be lost. The programs and components are highly unlikely to continue without donor assistance. Below, each program component is examined in terms of sustainability. 1: Participating youth acquire life and vocational skills, and knowledge on micro-enterprises. Interviews with participants demonstrated that after having completed the program two years ago, the youth interviewed are continuing their education, working in legal jobs and continuing to meet with their training colleagues to support one another. 2: Participating youth gain skills through vocational training that will allow them to enter the labor market. IDDI is using some of the youth as paid promoters in a health program they are implementing. The facilitators who were trained voiced hope that they could continue as facilitators in a new program. Interviews with former beneficiaries suggest that they continue to study, obtain higher degrees and find jobs. 3: Favorably improve gender equity and reproductive health knowledge of the target population Young women interviewed told the evaluation team that they “have learned how to negotiate with men” and have learned the take care of themselves. A study by the IABD has shown that there is an inverse relation between teen-age pregnancy and the 4-month Youth and Employment Program implemented by the Ministry of Labor and INFOTEP2 . It is possible that the similarly structured 8-month EpE program could have similar or better results. Recommendation(s) As the infrastructure (human resources and local institutions willing to collaborate) remains, future USAID programming focused on At Risk Youth could consider targeting at-risk youth in the Cotuí region and could knock on the door of Barrick Gold Mine Co. once again. 2 Domínguez, Luis, Evaluación cualitativa y de implementación para la evaluación de impacto del Programa de Juventud y Empleo (PJE) en República Dominicana: Informe Final, June 2011, p. 30 AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 10 3.4 Counterpart International Partner Program Summary Counterpart International (CPI) $125,000 10/15/08-10/14/10 La Romana-Bayahibe The overall goal of the Youth IT Training, Mentoring, and Job Placement for the Tourism Sector is to prepare disadvantaged young people for long-term careers in the hospitality and tourism sector by replicating a targeted, industry-focused technology tourism career development pilot program in the La Romana-Bayahibe Tourism Destination. Objectives (i) Contribute to the ongoing efforts of developing a national tourism workforce development program with special emphasis on assisting out-of-school youth; (ii) Define and implement new competency-based employability skills standards and build upon existing ones by strengthening the capacity of INFOTEP service providers to provide said skills to out-of-school youth interested in working in the tourism sector; and (iii) building capacity among employers to provide mentored work-based learning experiences that lead to long-term careers, for disadvantaged youth. This program provided disadvantaged youth with vocational training through agreements with INFOTEP and English language training by a local ESL private institution. Youth ages 17-25 received vocational training in tourism-sector jobs in the areas of lodging, food and beverage, travel and tourism, and IT. They also were placed in mentored internships in hotels. Children 7-15, including children in orphanages, received informal IT training, and families received workshops on issues relating to the problems faced by youth such as HIV, drugs, alcohol, violence and conflict. 156 youth ages 17-25 benefited from formal vocational training; only 9 left the program prior to completion. 103 youth participated in mentored work-based learning experiences, 44 were employed through a combination of full time and seasonal work with monthly earnings ranging from $650 to $850. 135 youth continued schooling following completion of training. 105 children/youth ages 7-15 directly benefited from informal IT training. 7 lead private sector firms provided youth training and employment opportunities for 88 youth. 15 institutions had increased understanding and capacity to address youth development, including government agencies, private firms, schools and community based organizations. In addition, through a series of workshops, 158 families increased awareness of the challenges youth encounter in their daily lives, such as alcohol and drug abuse, human trafficking, violence and conflict. Performance 94% of the participants who enrolled in the program completed and graduated. With regards to the first component, the project exceeded the objectives by increasing the number of participants from 75 to 156. With regards to the second component, it is unclear whether INFOTEP service providers were strengthened, however, many adjustments were made to the vocational training program, including raising the number of hours of training from 400 to as high as 900. 1,500 hours of English training were added after CPI found that hoteliers required basic English competency. With regards to the third component the project partially met the objectives as 44 participants were mentored and employed. The implementers were also able to leverage US$194,700 in cash and in-kind support. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 11 Efficiency The project as a whole was not operated very efficiently. An assessment of potential and available jobs was not undertaken before training was initiated. This is a best practice in employability programs for youth.3 Secondly firm commitments from hotels to provide mentored internships was not obtained beforehand (this could have been linked to the assessment, i.e. identify potential and available jobs in the region’s hotels and sign agreements to provide the training on the part of the project, and internships on the part of the hotels). The design should have included a prior needs assessment and solid agreements with potential employers to provide mentored internships. The needs assessment might have detected the need for ESL training, and the project should have organized this training prior to launch. Project documents report that CPI should have screened the youth better in terms of basic Spanish reading, writing and math ability and built in special training for those with these academic gaps. Sustainability The program is highly unlikely to continue without USAID assistance. The youth vocational training component is not sustainable as structured because INFOTEP was unable to certify the orphanage as an official training site. Future programs focused on at risk youth should review the CPI training program if training for tourism is to be offered in the new project. Recommendation(s) 1. An assessment should have been undertaken to better understand market demands before the program was structured and designed. 3.5 Alfalit Grant Partner Program Summary Alfalit $55,000 San Juan de la Maguana & Elías Piña Provinces October 2008-October 2009 2,500 adolescents/adults were to receive 160 hours of math and reading/writing and life skills. Objectives • Establish 50 centers with 1,250 participants per semester = 2,500 during LOP. • Train 7 coordinators and 90 facilitators. • It was expected that 80-85% would complete and 60% would continue education. Alfalit’s literacy program is designed for youth and adults ages 15-20 and includes reading, writing and math, and encourages participants to continue their education. Reading materials contain information on nutrition, health care, HIV prevention, home finances and life skills designed to improve everyday life of the family and the community. There were 65 Learning Centers, in the first period, located in private homes, classrooms, meeting rooms made available by institutions, and some in open-air settings. Performance According to the progress report of March 2009, 885 participants had been enrolled in San Juan de la Maguana province, of those 30 (3.4%) dropped out. In order to reach the goal 2,500 participants they 3 See: whatworks.uwex.edu/attachment/whatworks_09.pdf AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 12 would need to enroll an additional 1615 persons in the second semester. To achieve the additional enrollment the project (in March 2009) needed to recruit 110 additional facilitators. A report from a prior project (USAID 2005-2007) noted that 70% of the participants continued in school and obtained a basic education diploma. Others opted for vocational training, a few graduated from secondary Education. Efficiency No findings or conclusions are available on efficiency, because of the lack of project documentation. Sustainability Alfalit has continued to implement its programs without USAID assistance. Alfalit has for years implemented its programs in the DR and will continue to implement programs with or without USAID assistance as they have their own funds and other donors. 3.6 Major League Baseball-Dominican Development Alliance Partner Program Summary Major League Baseball (MLB) – Dominican Development Alliance (DDA) $1,000,000 Grant May 2009--‐ September 2012 USAID and MLB signed an alliance with six Dominican NGOs to implement rapid-response initiatives to benefit poor communities in the Dominican Republic. The purpose of the Alliance was to use baseball as a catalyst to fund sustainable development activities in order to improve the quality of life in Dominican communities by encouraging donations to the MLB-DDA from MLB teams, players and fans as well as supporters of the various members of the DDA, and to create broader intercultural understanding between Dominican and American communities with baseball as the medium for dialogue and exchange. Objectives • Fund sustainable development activities in order to improve the quality of life in Dominican communities. • Encourage donations to the MLB-DDA from MLB teams, players and fans, as well as supporters of the various members of the DDA. • Create a broader intercultural understanding between Dominican and American communities with baseball as the medium for dialogue and exchange. • Forge a long-term relationship between the six MLB-DDA partners and MLB with a shared vision of improving the lives of Dominicans. • MLB and/or its corporate sponsors take over funding of the MLB-DDA incentive fund beyond the three years of the USAID award. A total of 18 activities were funded by the MLB-DDA consortium. These USAID co-funded activities were implemented in Santo Domingo, Santiago, Higuey, San Pedro Macoris, Barahona, Paraíso, Najayo, Estorga, Caobete, Anamuya, and Manoguayabo. As of March 30, 2012, US$1,921,275 had been leveraged from MLB teams, players and fans as well as various members of the DDA. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 13 The project endeavored to incentivize actors from the baseball industry (i.e. players, teams, etc.) to donate to sustainable development projects implemented by NGOs in the DR. USAID contributed US$1million to establish the Incentive Fund that has functioned as a matching grants program that matches up to US$50,000 to approved community development projects. USAID committed US$669,663.58 of the US$775,720 available for subawards. The MLB channeled US$1,292,803.64 through its teams, players, fans or the MLB office and $628,471.79 through non-industry partners as the counterpart to the USAID Incentive fund. The projects included programming in health (diabetes, hygiene, HIV/AIDS prevention and treatment, mal nutrition), economic development through micro finance, education through support to schools and afterschool programs and youth development with a focus on at-risk youth. Each of the 18 projects were implemented as a stand-alone program by either World Vision, Save the Children Dominican Republic, and the Dominican Institute for Integrated Development (IDDI). One of the non-profit partners actively or passively implemented the projects. Passive implementation denotes those projects that are directly implemented by a non-member non-profit organization under the supervision of a member non-profit. IDDI has assumed all passively implemented projects. The beneficiaries of each project differed based on the wide range of projects that were included under the incentive fund. In general, the 25,370 adults who benefited are teachers that have been trained, community members that have improved access to medical care, or recipients of micro finance loans. The 13,177 children include students benefiting from education programs (either in or outside of school), at risk youth, and children of all ages benefiting from improved access to health care. Performance The evaluation team was not able to visit any of the 18 project sites to evaluate whether each of the activities had met targets. The Team was unable to review project reports, only the quarterly reports submitted to USAID by IDDI, which did not provide enough details on the projects to be able to discern what were the intended results or progress to date of each project. The project was able to leverage a total of US$1,927,560 from teams, players and fans through the third quarter of 2012. During the LOP the project received donations of equipment and visits by students from USA universities. A team owner has visited sites/projects she is co-financing. The DDA-NGOs are willing to continue to manage implementation of programs and share a vision of improving the lives of Dominicans. Negotiations are still underway to determine how the program can be restructured and continued. Team players, owners and managers have visited project sites. Students from several USA universities visited project sites and intermingled with project beneficiaries; reports were posted on blogs in the USA. Players and Fans contributed equipment and other goods that were either sent or brought along and distributed to participants or communities. The DDA partners are waiting to see if the program will be continued. The DDA partners are continuing to develop a strategy to continue the program. Efficiency IDDI very efficiently managed the 18 projects as it was able to employ a highly competent program/grants manager and had the backing of two advisors (John Seibel and David Luther) with significant experience in managing USAID projects and MLB activities in the DR. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 14 This program is comprised of dozens of programs, strategies, and interventions, which made it very difficult to determine whether the quality of life in Dominican communities has been improved due to interventions, and, made it difficult to efficiently manage. In order to take advantage of USA tax laws the MLB donors must donate to a USA-based charity which can then fund overseas programs. IDDI, Save the Children, World Vision and other international NGOs that operate in the DR are all registered charitable organizations in the USA. However, the MLB players also have a registered USA charitable foundation to which Dominican players can channel funds. Funds being channeled through multiple and/or different charities risk being directed toward multiple overhead and administrative costs associated with having one NGO subcontract or subgrant to another NGO. Sustainability The program is somewhat likely to continue without USAID assistance. At this point, it is not a question of more time for the original project but rather reaching an agreement with MLB/MLBPA to restructure the program so that it is more in line with their interests and can eventually become a program managed by them without the involvement of USAID. The lead implementing entity, IDDI, has prepared a concept paper that outlines a proposal for restructuring and continuing the program. Recommendation(s) The evaluation team was unable to determine whether funds can be channeled from the MLB players foundation for specific use in the DR (i.e. to one or more DR-based NGOs that would implement agreed￾upon programs). If teams, fans and others, as well as Dominican players can channel funds through the MLB players foundation, or for that matter, any one other foundation in the USA, to one or more NGOs in the DR that would manage implementation of programs this might be the most efficient way to make sure that the greatest percentage of the donations improves the quality of life in Dominican communities. The MLB donors could form a standing committee to monitor and evaluate the projects. In hindsight the design might have contemplated an agreement among funders to implement a small set of programs (e.g. strengthening community health programs, EpC programs for children, EpE programs for youth-at-risk, supporting PEF and Math programs in schools), and use/expand tried and tested programs that have been developed in the DR. A more thorough evaluation is necessary to ascertain to what extent life in targeted communities was improved. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 15 3.7 Batey Community Development Project Partner Program Summary Save the Children Federation Inc. with Implementing Partners Mujeres en Desarrollo Dominicana (MUDE) and Fundación para el Desarrollo Comunitario Save the Children Dominicana, Inc. (SCRD) $9,557,604.08 total grant November 4, 2008 – June 30, 2013 The education component of the project aims to improve access to high-quality education and extra￾curricular activities. The program places emphasis on the early grades (1-4) to ensure that children have a positive and successful first experience in school with the goal of increasing school attendance, performance, and primary school completion rates. Objectives (Components) 1. Increased access to primary school education through school infrastructure improvements and equipment and/or materials. 2. Improved quality of primary school education through enhanced teaching quality and administrator management capacity. 3. Improved enrollment and attendance of children in grades 1-4 4. Children and youth have access to and use extracurricular activities and groups which complement the formal curriculum Save the Children DR, a subsidiary of Save the Children Federation manages the education component of the Batey Community Development Project (BCD). This project aims to implement 4 components, listed above, in 12 Bateys with 8 schools in Hato Mayor and San Pedro Macoris Provinces. A component coordinator and two field staff oversee implementation of technical activities of the Components 2, 3 and 4, (which really act as sub-components of Component 1) and coordinate with Component 1 on all infrastructure work on school properties. The field staff visit schools, conduct training with the parent-community associations (APMAES), and coordinate school personnel trainings and the organization of EpCs and vocational training for youth and adults provided by ENTRENA and INFOTEP. While teacher training was originally provided through PUCMM, this was cancelled after preliminary tests showed little or no positive results. Performance Performance on each of the above listed components includes: 1. Infrastructure- All proposed infrastructure improvements, equipment and materials have been provided for the 8 schools that the project has targeted. Wells, pumps, roof-top water storage tanks, flush toilets, in some cases latrines, sinks have been installed in schools. Classrooms and entire schools have been remodeled/refurbished, and school grounds improved. Many of the wells and water systems provide water not only to schools but the homes as well. The schools are now attractive and safe places for children. Directors have a computer and printer/scanner/copier. Teachers and children have books and some classroom materials. Overall, the project is on track with respect to the first component in providing infrastructure, equipment and materials. 2. Improved quality of primary school through enhanced teacher and administrator capacity￾Teachers have received training in methodologies and for teaching reading in grades 1-4. They also have received 2-days of Quantum Learning training but no follow-on training nor coaching. The evaluation team did not receive any documentation that describes the results expected from AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 16 the QL. No data has been collected that could be used to measure the impact of the Quantum Learning methodology training. School Directors are participating a MOE specialized training program, the Modelo de Gestión (Management Model) program but similar to the teacher training referenced above, the evaluation team did not receive any documentation that describes the results expected from the Modelo de Gestion programs and there is no data available that includes the Directors of the 8 Batey schools. Parent-community associations are receiving training from AMAPES in how to support schools and their children and are providing timely support to schools and to their children. The APMAES were created and established as legally recognized associations, with bylaws, rules and work plans, to be responsible, after the project is completed, to support schools, especially the maintenance of school property. The contract with PUCMM/ESP to provide additional training in literacy and math was cancelled. Overall, with regard to the second component, there is a lot of work to do since the contract with PUCMM/ESP was cancelled and no math training was offered. It is not clear whether the PUCMM training in reading and in math can be provided to these batey schools. There is still a need for follow-on training, or at least, coaching related to the Quantum Learning methodology. 3. Student Attendance Increased- According to the 2010 Program Description statement of (October 2010), attendance was 77% (target of 85%) and 90% had passed 4th grade (target 94%). Overall, school enrollment, attendance and 4th grade promotion seem to be on track. Interviews and documents suggest that this is likely due to the physical improvement of schools and that students have books and materials in the classrooms and school personnel have received some training. MINERD is providing a daily food ration to students, and this is reported as another factor that is likely increasing attendance. While enrollment and attendance rates have improved, only anecdotal evidence is available to tie this increase to project activities. 4. Students participate in out of school activities- According to the 2010 Program Description statement of (October 2010), 196 children ages 6-14 had enrolled in extra-curricular programs (target 275). Teachers reported to the AMEX team that they noted significant improvement in children who had participated in the EpC program. According to the 2010 Program Description statement of (October 2010) 273 youth ages 14-18 had participated in vocational or informal education programs (target 700). The EpC and EpE programs were implemented but were terminated at the end of the 2011-2012 school year, as funding for this activity ended. Table 1 shows the tracking of education component targets through September 2012. Efficiency While it appeared that component 1, focused on infrastructure, equipment and materials operated efficiently, capacity building for school personnel could have been managed more efficiently. The original design could have contemplated a more thorough diagnosis to ascertain the type of training that school personnel in batey schools need, and how the training should be implemented. It was evident to the AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 17 evaluation team that more accompaniment or clinical supervision is needed. Sporadic training is not enough; these teachers could benefit from a more intensive program, especially in how to teach reading and math. While the Quantum Learning methodology has demonstrated success in other settings, it requires more training days and more coaching visits. The management team could have benefited by having a full-time education specialist to regularly visit schools and work with directors and teachers. Implementing the ESP program within the BCD might have benefited in working closely with a project education specialist who would be in the schools to support the PEF trainers. Sustainability It is somewhat likely that the APMAES will continue to function and provide support to the schools, especially if Save the Children continues to work in the Bateys with other funding, after USAID assistance terminates. It is somewhat unlikely that Quantum Learning will continue or PEF training will be resumed as funding for these initiatives is uncertain. It is somewhat likely that the enrollment and attendance rates will not decrease from the current levels as parents are much more aware about the importance of schooling through project training, schools are more attractive, school personnel are better trained, the school food program is functioning well and the APMAES are organized The EpC and EpE programs require ongoing funding. Both programs were terminated at the close of the 2011-12 school year. During interviews, teachers and directors have expressed their hope that the EpC and EpE programs be continued as they have been instrumental in helping children perform better personally, socially and academically. Key informant interviews with MINERD revelaed that funds may be available to underwrite the EpC program. The project staff should consult with the MINERD Office of International Cooperation. Recommendation Given the relative success of components 1,3 and 4 and the challenges faced in component 2, the evaluation team recommends placing a significant focus on achieving component 2 in the remaining months of the program. During the final period of the project, in the second semester of the current school year, a more intensive QL training could be provided for teachers. The project could also work with the MOE in strengthening the training program for the principals, through increasing coaching/clinical supervision visits to the schools by qualified trainer/coaches. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 17 Appendix D-1: Bibliography Alfalit International Inc., Youth and Adult Literacy Project. Progress Report October 2009-March 2009. Alfalit International Inc., Youth and Adult Literacy Project. Award Letter. (October 15, 2008) Alfalit International Inc., Youth and Adult Literacy Project. Alfabetizacion sin limites. Counterpart International. Training, Mentoring, and Job Placement for Disadvantaged Youth in the DR. Quarterly Reports 1,2, & 4. Counterpart International. Training, Mentoring, and Job Placement for Disadvantaged Youth in the DR. Final Report. November, 14, 2010. USAID. Training, Mentoring, and Job Placement for Disadvantaged Youth in the DR. Award Letter & Cooperative Agreement. October 2008. CRS. Project Learning Together. A Community-Based Approach to Provide Educational Opportunities to Children and Youngsters Out Of The School System. Reference: No. 517-08-001. Final Narrative Report. October 2010. CRS. Project Learning Together II. A Community-Based Approach to Provide Educational Opportunities to Children and Adolescents Out Of The School System. Final Narrative Report. September 2011. CRS. Project Learning Together. A Community-Based Approach to Provide Educational Opportunities to Out of School and At Risk Children and Youth. Award Letter. (May, 23, 2008) DREAM Project. Five Year Report. 2007. DREAM Project. Annual Report. 2008-2009. DREAM. Performance Report Guzmán Ariza DREAM Summer School Camp 2008. October 2008. DREAM. Performance Report Guzmán Ariza DREAM Summer School Camp 2008. February 2009. DREAM. Performance Report Guzmán Ariza DREAM Summer School Camp 2008. May 2009. USAID. Guzmán Ariza DREAM Summer School Camp. Award letter and Cooperative Agreement. April 2008. IDDI. Lista de Cursos en Processo. Centro de Capacitacion Politecnico La Immaculada. IDDI. Lic. Silvia Viñals, Dra. Juliana Fajardo. Informe Evaluación Intermedia Proyecto. “Reinserción Escolar y Fomento a la Incorporación Productiva de Jóvenes de Cotuí”. (September 2010) IDDI. School Reinsertion and Promotion of Productive Development of Youth in Cotui. Project Monitoring Report. (Reports 1-4) AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex D: ARCY 18 IDDI. Reinserción Escolar y Fomento a la Incorporación Productiva a Jóvenes de 13 A 21 años Residentes en Sectores del Municipio de Cotui, Provincia Sánchez Ramírez. Informe Narrativo Primer Año. December 2009. IDDI. “School Reinsertion and Promotion of Productive Development of Youth Aged 13 to 21 years, residents of the Municipality of Cotui, Sanchez Ramirez Province.” Award Letter & Cooperative Agreement. (May 20, 2008.) MLB-DDA. Attachment B- Program Description. MLB-DDA. Project Overview. October 2011. MLB-DDA Field Trip Itinerary. August 2011. MLB-DDA. M&E Site Visit. March 1, 2011; March 16, 2011; March 24, 2011; Aug 31, 2011. MLB-DDA. Visita a Proyectos. Aug 31, 2011. MLB-DDA. Quarterly Reports. Quarters 1-4, 2011, Quarters 1 – 3(Final Report), 2012. MLB-DDA. Projects Funded through the MLB-DDA Incentive Fund. MLB-DDA. USAID support to Batey Communities through the MLB-DDA program. Save the Children. Batey Community Development Project. Quarterly Reports. July-Sept 2010; Oct Dec 2010; Apr-June 2011; July-Sept 2011; Jan-Mar 2012; April-June 2012. Save the Children. Batey Community Development Project. Federal Financial Report. Dec 31, 2010. Save the Children. Batey Community Development Project. Modification of Assistance. Save the Children. Batey Community Development Project. Annual Performance Report. FY 2011. Annex E: Individuals Consulted Submitted to: USAID/Dominican Republic January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Table 1: Data Collected (Schools, Director and Teacher Interviews, and Classroom Observations) ESP BEP BATEY TOTAL School Centers 24 10 2 36 Director Interviews 24 11 2 37 Teacher Interviews 28 18 3 49 Classroom Observations 42 15 3 60 Group interviews: 3 groups of parents 4 groups of youth (including Safe Schools) Table 2: Key Informant Interviews (including representatives from MINERD, private sector, and implementing partners Organization Person/Title When ESP BEP BCD/Ed AYRC USAID Kevin Roberts/Education Officer Sept 28 Oct 2, 3, 4 √ √ √ √ √ USAID Maricela Ramirez/Program Development Officer Oct 2, 4 √ √ √ √ USAID Marina Tavarez Oct 2, Oct 3 Oct 16 √ USAID Josué Ceballos, USAID COR, MLB￾DDA Oct 4 √ Ministry of Education (MINERD) Minister of Education, Josefina Pimentel Oct 29 √ √ √ √ MINERD Rita Ceballos/ Director General of Basic Education Oct 29 √ √ √ √ MINERD Fernando Ogando/ Director Office of International Cooperation Oct 25 √ √ √ √ MINERD Ing. Francisco G. Estrella F./ Director General- Auditoría y Control de Procesos Educativos Oct. 3 √ MINERD Ancell Scheker Mendoza, PhD./ Director General- Evaluacion y Control de la Calidad de la Educacion Oct. 4 √ Ministry of Education￾District level Técnica del Distrito 0803 Ecargado de Planificacion, Distrito 1503 Oct 18 Oct 29 √ InterAmerican Development Bank Horacio Alvarez Marinelli, Education Specialist Oct 29 √ Organizacion de Estados Iberoamericanos Catalina Andujar Schekler, Representante Permanente de la OEI Oct 10 √ Fundacion Inicia Angela Español, Gerente de Projectos Oct 12 √ ISFODOSA Ana Dolores Guzmán, Directora Ejecutiva Oct 12 √ Directora del Centro de Excelencia para la Investigación y Difusión de la Lectura y Escritura (CEDILE) Dra. Liliana Montenegro Oct 22 √ AES Dominicana Tamara Navarro Oct 10 √ Organization Person/Title When ESP BEP BCD/Ed AYRC AMCHAM Liliana Cruz Oct 2 Oct 10 Oct 28 √ AMCHAM Juana Díaz Oct 3 Oct 5 Oct 28 √ IMCA Dr. Pedro Esteva, President and CEO Oct 26 √ Ediciones SM Monica Volonteri Oct 8 √ EDUCA Aida Consuelo Hernandez Executive Vice President Oct 4 Oct 8 √ √ √ EDUCA Hernan Vélez, EpC/EpE Coordinator Oct 15 √ √ EDUCA Liliana Degiorgis, Researcher Oct 2 √ ENTRENA John Seibel Oct 3 √ √ ENTRENA Saschia Seibel, Co-Coordinator Cotuí Oct 10 √ √ ENTRENA Trainer (Esther) Oct 8 Oct 20 √ √ Fundación Rica Wenceslao Soto, Sponsor Representative Oct 12 √ INOFOCAM Denia Burgos, Directora Ejucativa Oct 12 Instituto Superior de Formación Docente Salome Urena (ISFODOSO) Ana Dolores Guzman/ Rectora TBD √ √ Pontificia Universidad Católica Madre y Maestra (PUCMM) Sarah González de Lora Vicerrectora de Investigación e Innovación Oct. 3, Oct 29 √ PUCMM Gineda Castillo, previous Directora Ejucativa INOFOCAM, pending Dir. Edu Mgt Oct 8 √ √ PUCMM Gilberto Valverde, Sr. Researcher, Academic Chair, Director of Evaluation Component SUNY-Albany Oct 17 √ PUCMM Jose Alberto Contreras, Tania Jimenez Rosa, tecnicos Education Management Oct. 5 √ PUCMM Romona Santana (4yrs), Technical Assistance of Padres y Madres en Acción Oct 5 √ Organization Person/Title When ESP BEP BCD/Ed AYRC PUCMM Barbara Campos, Sonya Mendezñ Coordinadores de MAT Oct 8 √ PUCMM Marys Encarnacion, Jose Anibal Paula Ceballos, Denia Feliz Gomez, Dulce Maria del Rosario ; Accompanantes de Gestión Oct 8 √ PUCMM Sarah Guilamo, Directora, Escuelas Seguras Oct 12 √ PUCMM Victoria Rosado (2 years), Daily Yajahira Perez Salvador (5 years), Ariel Pacheco; Alta Gracia Herrera (3 years); Capacitadoras- LE Oct 12 √ PUCMM Eleuterio Martinez, Accompante Matematicas Oct 17 √ Batey Community Development Project (Save the Children) Becky Myton, COP and Aneliya Nikolova, Education Component Coordinator Oct 2 √ IDDI David Luther, President Oct 8 √ CRS Erick Gómez, Project Coordinator Oct 5 √ CRS Kellie Hynes, Country Representative TBD √ IDDI Liliana Rocha, Co-Coordinator Cotui Oct 4 √ √ IDDI María Almonte, Field Coordinator Cotuí Oct 13 √ IDDI Michael Rosario,Health Coordinator Cotuí Oct 13 √ MLB-DDA - IDDI Pamela Schreier, Program Coordinator Oct 8 √ CRS Stephanie Frias,Coordinator Oct 5 √ Caminantes/CRS Yolanda, Psychologist Oct 5 √ DREAM Catherine DeLaura, Executive Director and Molly Oct 24 √ Instituto Tecnológco Superior Comunitario (ITSC) Victor Hugo de Lancer, Rector a.i Oct 20 √ ITSC Arq. Julio Peña, Vice-Réctor a.i. Oct 20 √ AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex F: Methodology i Annex F: Methodology Submitted to: USAID/Dominican Republic January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex F: Methodology ii The mid-term performance evaluation of the education portfolio has two principal objectives: to identify mid-course corrections for the Effective Schools and Basic Education projects; and, to inform the Mission’s new education strategy in setting priorities among programs and components for cost-effective use of Mission resources. Annex F presents the overall evaluation questions, the data sources and instruments, and the specifics of the data collection and analysis process by project. Evaluation Questions As outlined in section 1.2 of the evaluation report, the evaluation is organized around three broad questions, which are applied first to each of the projects and then at the level of the education portfolio as a whole. The definition of these questions at the project level is: (1) Performance. The extent to which the project is on track to produce the expected results, and the contribution of these results to improved quality of basic education. (2) Efficiency. The utilization of resources relative to results. The analysis of efficiency is intended to identify core aspects of the projects that are essential to the overall objective. In this case, the evaluation is not intended to analyze budget or cost data. What changes could be made in the implementation of the project so that it operates more smoothly and effectively? (3) Sustainability. Various factors will affect whether the project could continue without USAID assistance. The mid-term evaluation examines these factors and recommends adjustments that will increase the probability of sustainability. At the portfolio level these questions are applied to factors that cut across the individual programs to define the USAID/DR education portfolio as a whole. (1) Performance. - Program components: USAID/DR has identified deficiencies that contribute to poor quality in education in four areas – teacher effectiveness, community participation, resources, and policy. Common components to address these deficiencies are present in multiple projects. The integrated evaluation examines the relative performance of these components across projects. - Gender equality: According to USAID policy, all activities should support gender equality, and the USAID/DR education strategy states that boys and girls should benefit equally. The integrated evaluation examines attention to gender considerations in the portfolio. - Performance monitoring: Monitoring project indicators throughout the life of a project is the basis for tracking whether the activities are on track. Monitoring data also is essential input to project evaluations. With baseline measures, outcome indicators measure the degree of change in key project results. Indicators also may provide a basis for assessing the contribution of a particular activity to program results. The integrated evaluation discusses monitoring as a tool in assessing the USAID/DR education portfolio. (2) Efficiency. The integrated evaluation raises two questions about efficiency in the portfolio as a whole: (1) What are the relative benefits of repeating various components across projects, as opposed for example to specialization; and (2) To what extent do the projects in the portfolio form an integrated program? Is the effect of the package greater than the individual activities? (3) Sustainability. Questions about sustainability related to project components examine the extent to which the USAID/DR education portfolio contributes to improved quality of education in the country as a whole? Is the coordination between the USAID/DR education portfolio and other key actors in the system (especially MINERD, NGOs, private sector) sufficient to impact the system AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex F: Methodology iii through development of pilot programs or models for activities to counter the systemic deficiencies and to provide a basis for institutionalization of the programs without donor funding? Data Sources and Instruments The evaluation team employed a mixed methods approach to examine the programs and components that make up the USAID/DR education portfolio and to address the questions listed above. The evaluation team relied primarily on the sources of data listed below. Document Review: USAID/DR, implementing partners, and other stakeholders provided available records and reports for each of the five programs. These documents included program descriptions, Performance Management Plans (PMPs), Quarterly and Annual Reports, project materials, presentations and publicity materials about projects, Annual Work Plans, and in some cases, external evaluations or assessments of projects or project components. In some cases requested documents were not available. Interviews (Key Informant and Small Group): The evaluation also drew on self-reported experiences and opinions solicited through key informant and small group interviews with program direct and indirect beneficiaries (teachers, school administrators, parents, community leaders, and in some cases children and youth). The evaluation team utilized standardized evaluation instruments (key informant interviews, teacher and director interviews) to collect data on the effects of these programs on the individual’s ability to provide and receive better education services. The Team conducted interviews, both individually with particular stakeholders and in small group settings, to provide qualitative and quantitative data that was triangulated with other data and sources. Interviews with both men and women were sought in each category. Classroom Observation: Team AMEX collected data on teacher’s instructional methods and student participation by utilizing classroom observations in randomly selected beneficiary schools. The methodology utilized to select the schools is listed in the next section. The instruments used to conduct interviews and classroom observation are appended to this annex. Data Collection by Project While all data collection for each project relied on the above sources and instruments, application of the evaluation questions, school site selection, and the focus of data sources varied by project. This section highlights the methodology used to collect and analyze data for each project. Effective Schools Program: With the data base provided by USAID of ESP program sites, schools were identified as PEF-USAID. These schools were then cross referenced with each of the teacher training intervention data bases (Mathematics and Spanish Language) to ensure that schools selected had an equal distribution of each intervention. Other schools in the data base provided by USAID were identified as PEF-MINERD; they receive technical assistance from PUCMM to expand the model with MINERD funds. See table 1 for distribution of schools (“Comprehensive” indicates both Mathematics and Spanish interventions.) In the priority areas identified, samples were selected randomly to be (1) representative of the overall distributions by region and at the same time (2) to include important program components we wanted to observe. 1 The sample was checked to ensure representation of the Mathematics and Spanish 1 Randomly selected to the extent possible with Data Analysis and reflects case selection described by Seawright & Gerring, 2008 (typical and diverse); the danger in randomly selecting cases when sample sizes are small is that they tend to be unrepresentative of the population, we used typical and diverse methods to select our sample. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex F: Methodology iv Language components, the Education Management component, the Safe Schools program, and years of participation in the program (when available). See table 2 for distribution by year. Table 1. Observation distribution, n=42- ESP, by treatment Row Labels Count of Escuela % Comprehensive (both) 9 21% at least Mathematics 21 50% at least Spanish 25 60% Table 2. Distribution of classroom observations - ESP, by year incorporated (according to database) Year # of Schools % 2006 5 12% 2007 5 12% 2008 8 19% 2009 13 31% UNAVALIABLE 11 26% Grand Total 42 100% For the ESP project, data was collected through 30 key informant interviews, 42 classroom observations (30 minutes each on average), 24 director interviews, 28 teacher interviews, 3 group interviews with youth from the Safe Schools program and 3 group interviews with parents from the ESP schools (see Appendix A-3 for list of key informant interviews, and Appendix A-4 for a list of schools). Key informant interviews and group interviews were recorded and coded into general themes. Similarly, teacher and director interview responses were grouped by question and responses consolidated. Classroom observation data was tallied and percentages calculated; they were differentiated between duration in mathematics and literacy interventions when applicable. Also unique to the ESP methodology is the review of student achievement data from student assessments administered between 2008 and 2011. This data includes a control sample in 2010 and 2011. Analysis can be found in Annex B. Basic Education Program: During the document review for the BEP program, the evaluation team found that for BEP there is a variation in the intensity of interventions (number and types) by school. Based on this preliminary finding, the evaluation team proposed to select schools for site visits based on the number of interventions being implemented in the school. Selecting the schools in this way allowed the team to be able to conduct classroom observations in a variety of settings. As part of the planning process, both USAID and the evaluation team agreed that the intensity of the interventions may affect perception of the BEP program at the school level. Table 3 summarizes the distribution of the schools by intensity of intervention. AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance EvaluationBasic Education Program Annex F: Methodology v Utilizing AMCHAMDR records (list of 51 schools as of 3/30/2012), the team selected schools for classroom observation. The selection process was as follows: o Schools that were outside of a geographic focus area for USAID were removed from the list (43 schools remained). o For each type of intervention, the school was scored as 0 or 1. o The number of nterventions was summed by school and then grouped or stratified (column 1). o A software program (MS Excel Data Analysis Tools) was utilized to generate a random selection (column 2) for each grouping. The evaluation team scheduled visits to a total of 11 schools although one of the schools was closed due to an unexpected holiday (celebrations of the Teacher Union’s elections). The distribution of the schools visited (10) and classrooms (15) observed varied from the original design selection because the actual interventions received differed from the records reviewed. The gaps in records and timing are examined in the BEP Evaluation report (Annex C). Table 3. Classrooms Observed Column 1 Total Number of School per AMCHAMDR Records Column 2 Selected Schools Column 3 Schools/Classrooms Visited Four/Five Interventions 11 3 4/4 Three Interventions 10 3 4/8 Two Interventions 12 4 1/1 One or Zero Interventions 10 2 1/2 43 12 10/15 As the evaluation questions were finalized, in consultation with USAID, it was determined that two elements of the BEP program would be excluded from the evaluation. These are: • Recruitment of Private Sector Sponsors In the near term, AMCHAM/DR is conducting an assessment of the private sector partnership component; therefore, this evaluation does not assess the performance targets related to securing the number of partners outlined in the program description. • Educational Achievement. USAID is financing the application of a standardized measurement of student achievement (Early Grade Reading Measurement and Early Grade Mathematics Measurement) that was underway at the time of the field work for this evaluation. Thus, understanding performance for this element of the theory of change is not part of the evaluation. At Risk Children and Youth Programs and the Batey Community Development Program (Education Activities): Data specific to the evaluation of these projects relied primarily on review of documents provided by USAID and implementing partners, interviews with project managers and beneficiaries, and a few visits to project sites. Of the nine schools in the Batey Community Development Program the team randomly selected four for school visits. Beneficiary Interview Guide- DIRECTORS (Post Pilot-Final) Page 1 of 3 Name: Sex: Current school: Per Records Training Received Preguntas al/la Director(a) 1. ¿Cuánto tiempo tiene usted como docente? __________ años 2. ¿Cuánto tiempo tiene usted como director? __________ años 3. ¿Cuánto tiempo tiene usted en esta escuela? __________ años 4. ¿En qué tandas es usted director de esta escuela? ___ Matutina ___ Vespertina ___ Ambas 5. ¿Cuál es su nivel de formación?________________________________________________ 6. ¿Cuál es su especialización? __________________________________________________ 7. ¿Está usted estudiando actualmente? Sí _________ No _______ a. ¿Dónde? ___________________________________________________________ b. ¿Cuál carrera? Sobre La Escuela 1. ¿Tiene la escuela una subdirectora ______ / coordinadora del 1º Ciclo ____ / orientadora _____? 2. No de Aulas __________________ 3. No de Docentes _______________ ¿Cuántos docentes repiten en las dos tandas? __________ 4. Grados en la mañana: 5. Grados en la tarde: 6. ¿Hay un Asociación de Padres y Madres y Amigos de la Escuela activa? ___________ PREGUNTAS PARA INICIAR LA CONVERSACIÓN 1. ¿Cuáles son los programas, organizaciones o instituciones que han trabajado con la escuela en los últimos dos años escolares? Comencemos con el último año, 2011-2012: Proyectos 2011-2012 Proyectos 2010-2011 a. (Si no menciona algo de dotación de materiales, pregunte:) ¿Qué tipo de materiales ha recibido del Proyecto ___________? b. (Si no menciona algo de capacitación, pregunte:) ¿Qué tipo de capacitación ha recibido la escuela del Proyecto ___________? Beneficiary Interview Guide- DIRECTORS (Post Pilot-Final) Page 2 of 3 2. ¿En cuáles proyectos o programas usted ha participado, personalmente? [si no menciona ninguno de los nuestros, pasa a la pregunta 8] 3. ¿Cuándo participó usted en ese programa? _______________________ ¿Cuántas horas? ______________________________ 4. ¿Qué me puede comentar sobre las visitas de seguimiento que hace el programa a esta escuela? 5. ¿Le ha ayudado _____________? [para cada proyecto/programa] a. ¿Cómo? b. Si no, ¿por qué? 6. ¿Cuáles son las estrategias y metodología que les ha ofrecido el Proyecto? 7. ¿Cuáles son los cambios que usted lo ha visto después de que se inicio el proyecto _____________? [Use el nombre que ell@s emplearon arriba] PREGUNTAS SOBRE EL CENTRO 8. Sobre el aprendizaje de NN de esta escuela, en lectura, ¿cuáles cree usted que son los mayores logros? 9. ¿Y cuáles son las mayores dificultades, para el aprendizaje de la lectura? 10. Sobre el aprendizaje de NN de esta escuela, en matemática, ¿cuáles cree usted que son los mayores logros? 11. ¿Y cuáles son las mayores dificultades, para el aprendizaje de la matemática? 12. ¿Cuáles son los dos problemas más graves que afectan esta escuela? 13. ¿Y cuáles son los dos logros más importantes? Beneficiary Interview Guide- DIRECTORS (Post Pilot-Final) Page 3 of 3 14. ¿En qué emplea usted más tiempo, durante la semana, en la escuela? 15. ¿Qué haría usted, si pudiese disponer de más tiempo? 16. ¿Qué haría usted, si pudiese disponer de más recursos? Para BEP 17. ¿Cuál ha sido la participación de la empresa que patrocina esta escuela? 18. ¿Cuál es su percepción sobre cómo se ha manejado el programa? 19. ¿Cómo definieron las necesidades con la Cámara? Para PEF 1. ¿Qué tipo de apoyo ha recibido, específicamente en el área de gestión? 2. ¿Quiénes son los actores involucrados (si no menciona, preguntar si hay visitas de acompañamiento de los técnicos distritales)? 3. ¿Cuáles han sido los beneficios del apoyo que le han brindado? 4. ¿Qué tipo de apoyo en el área de gestión falta por brindar? Beneficiary Interview Guide- Teachers Page 1 of 2 Name: Sex: Current school: Per Records Training Received Preguntas A. ¿Cuánto tiempo tiene usted como docente? __________ años B. ¿Cuánto tiempo tiene usted en esta escuela? __________ años C. ¿En qué tanda y grado trabaja usted en esta escuela? __ Matutina __ Vespertina __ Ambas D. ¿Trabaja usted en otra escuela? ¿cual? ___________________________________________ PREGUNTAS PARA INICIAR LA CONVERSACIÓN 1. ¿Cuáles son los programas o instituciones que han trabajado con la escuela en los últimos dos años escolares? Comencemos con el último año pasado, 2011-2012: Proyectos en 2011-2012 Proyectos 2010-2011 a. (Si no menciona algo de dotación de materiales, pregunte:) ¿Qué tipo de materiales ha recibido del Proyecto ___________? b. (Si no menciona algo de capacitación, pregunte:) ¿Qué tipo de capacitación ha recibido la escuela del Proyecto ___________? 2. ¿En cuáles proyectos o programas usted ha participado, personalmente? [si no menciona ninguno de los nuestros, pasa a la pregunta 8] 3. ¿Cuándo participó usted en ese programa? _______________________ ¿Cuántas horas? ______________________________ 4. ¿Qué me puede comentar sobre las visitas de seguimiento que hace el programa? 5. ¿Le ha ayudado _____________? [para cada proyecto/programa] a. ¿Cómo? b. Si no, ¿por qué? 6. ¿Cuáles son las estrategias y metodología que les ha ofrecido el Proyecto _____________? Beneficiary Interview Guide- Teachers Page 2 of 2 7. ¿Cuáles son los cambios que usted lo ha visto después de que se inició el proyecto _____________? [Use el nombre que ell@s emplearon arriba] PREGUNTAS SOBRE EL CENTRO 8. Sobre el aprendizaje de NN de esta escuela, en lectura, ¿cuáles cree usted que son los mayores logros? 9. ¿Y cuáles son las mayores dificultades, para el aprendizaje de la lectura? 10. Sobre el aprendizaje de NN de esta escuela, en matemática, ¿cuáles cree usted que son los mayores logros? 11. ¿Y cuáles son las mayores dificultades, para el aprendizaje de la matemática? 12. ¿Cuáles son los dos problemas más graves que afectan esta escuela? 13. ¿Y cuáles son los dos logros más importantes? 14. ¿Qué haría usted, si pudiese disponer de más tiempo? 15. ¿Qué haría usted, si pudiese disponer de más recursos? Ultimas PREGUNTAS E. ¿Cuál es su nivel de formación?__________________________________________________ F. Cuál es su especialización? ____________________________________________________ CLASSROOM OBSERVATION-POST PILOT - FINAL Page 1 of 4 DATOS BÁSICOS Nombre del centro: Código único: Turno (M/V): Región: Distrito: Zona (rural/ urbano): Fecha (dd/mm/aa): Hora de inicia (hora/ min): Hora terminada: (hora/ min): Quien Observa: DATOS DE LA AULA Grado (1, 2, 3, 4): Apellido de maestro/a: Nombre de maestro/a: Área que esta enseñando (LE, MAT, otra): Tema de la clase: Número de estudiantes: Niñas __________ Niños __________ Total __________ Preguntas para el/la maestro/a: Años de experiencia: Nivel de educación más alto: PEF ¿Recibe o ha recibido el Componente LE (si/no)? ¿Recibe o ha recibido el Componente MAT (si/no)? BEP ¿Planificación y creatividad? ¿QL? ¿Animación en Lectura? ¿Otros? Fechas de capacitación: CLASSROOM OBSERVATION-POST PILOT - FINAL Page 2 of 4 CONTEXT MATERIALES/ AMBIENTACION 1. Teacher guide: _____ 1= the teacher has a teacher guide and has consulted it 2= the teacher has a teacher guide and does not consult 3= the teacher does not have a teacher guide Observations: 2. Teacher plan:_____ 1= the teacher has a lesson plan and refers to it heavily 2= the teacher has a lesson plan and refers to it on occasion 3= the teacher has a lesson plan, but does not use 4= the teacher does not have lesson plan Observations: 3. Student texts (math only): _____ 1= all the students have textbooks (100 percent) 2= more than half the students have textbooks (50- 99percent) 3= less than half the students have textbooks (0-49 percent) 4= there are no textbooks (0 percent) 3a. Students use the texts in the lesson (yes/no):_____ Observations 4. There is a biblioteca de aula (yes/ no): _____ Note: the biblioteca de aula is a donation under the PEF program of 70 books at the grade level of the class. BEP has donated materials to MOST schools. 4a. About how many books are in the biblioteca? _____ 4b. Students use the biblioteca de aula in the lesson (yes/no)? ______ Observations: 5. There is a caja de matematicas in the aula (yes/no): ______ Note: the caja de matematicas in the PEF program is a plastic box that has various mathematics materials, “bloques de diez”, clocks, fichas, etc.; there is one per 4 grades in a school. 6. There other materials in the classroom (yes/no): ____ Note: This could be education materials from a store, recycled materials from home, additional books other than the biblioteca. 6a.What are they?: 6b. Are they being used in the lesson (yes/no): ______ 7. Classroom education materiales created by students and teachers on the wall ____ 1= the walls are covered with education related materials that are age appropriate (this implies that they are inviting to the students, i.e. colorful) 2= the walls have some education materials, but they may not be age appropriate or not accessible to students in the class, or they may not be education related 3= there is little to no education related material son the walls 7a. How many of the walls available are covered?____ Observations: CLASSROOM OBSERVATION-POST PILOT - FINAL Page 3 of 4 8. There are rincones de aprendizaje in the classroom (yes/no) _____ Note: a rincón de aprendizaje is a small space that is established where various resources are established for students to work independently or in small groups and learn about a particular theme, there may be books, materials, posters, etc. 8a. How many? ____ 8b. What area?______ 8c. Are students using it:____ Observations: 9. How are the students seated (draw on right, identify teacher)?_______ 10. How are the students working in the class____ a= the teacher stands at the front of the class and the students are in their seats b= students are in pairs working c= the students are in small groups working d=not working 11. What is the percentage of students engaged in learning activities?____ 1= all the students are engaged (100 percent) 2= over half the students are engaged (50-99 percent) 3= less than half the students are engaged (0-49 percent) 4= none of the students are engaged (0 percent) Observations: CLASSROOM PROCESSES 12. Active teaching methodology/ constructivist approach Note: an active teaching-learning methodology as defined by the program entails reflection on students previous knowledge, constructing knowledge actively using materials, reviewing what is learned and applying the knowledge. 12a. the teacher engages students in a dialogue (yes/no): _____ 12b. students demonstrate knowledge with materials (yes/no):_____ 12c. students have opportunities to practice what they learn (yes/no):_____ 12c. the students apply what they are learning (yes/no): ____ Observations: FRONT BACK CLASSROOM OBSERVATION-POST PILOT - FINAL Page 4 of 4 13. The teacher uses the following techniques (observations to right): The teacher 13a. the teacher writes on the board To provoke a discussion____ To have students answer questions_____ To have students copy_____ To summarize____ To have students read_____ 13b. the teacher verbally Dictates____ Reads aloud ____ Asks students questions (individually)____ Asks students questions (as a group) _____ 13c. the teacher physically demonstrates_____ 13d. the teacher physically demonstrates with materials____ 13e. the teacher draws___ 13f. the teacher uses a poster____ The students 13g. the students read Aloud____ Silently____ 13h. the students write on the board ____ 13i. the students role play____ 13k. the students demonstrate physically_____ 13l. the students demonstrate with materials_____ 13m. the students do seatwork_____ 13n. other _____ 14. Reinforcement techniques (choose all that may apply): 14a.The teacher explicitly thanks students for participation____ 14b. The teacher reinforces with a positive expression_____ 14c. The teacher is neutral____ 14d. The teacher criticizes individual students_____ 14e. The teacher criticizes the whole class____ 15. The teacher moves to address students’ discipline needs (yes/ no)____ 16. The teacher moves to address students’ learning needs (yes/no) _____ 17. Gender analysis, girls and boys participation______ FOR FIVE MINS Note: count the number of times the teacher calls on girls/ boys in the table below 1= girls and boys are given fairly even opportunities to participate 2= girls are given more opportunities to participate 3= boys are given more opportunities to participate 3= there is little to no participation Girls Boys Observations: Interview Protocol for the Key Informants from the Implementer Provide description of the evaluation Advise that attribution is likely. Name Title/Position How long with the project? Role Programmatic Please describe the project purpose. Please describe progress to date for the various components. Please describe who are the beneficiaries? How are the needs assessed? How are the interventions designed/selected? How do you describe success? How have you interacted With the government? At what level? With the private sector? With the donor? How is the project incorporating gender analysis? Copy of the gender strategy/analysis? Please describe your process for monitoring progress? Are there specific instruments? What are some of the assessments and other type of analysis that have been completed? What are some of the planned evaluations, assessments, etc? Operational/Efficiency Questions - Internal What component of the project has been easiest to implement, administratively? What has been the most time consuming? If you can think about how you spend your time in a month, what is the % of time Name on: - Reporting - Technical review - Financial If you had more time, what would be your top priority activity? Second? If you had more funds, what would be your top priority activity? Second? Are there areas in which you might receive more training? Operational Efficiency - External Are there are programs that are similar? Do you meet/coordinate with other implementers? What would help you to implement the objectives of the project? Sustainability Which activities should be sustained but face what difficulties? What is needed in order to overcome the/those difficulties? Does the government see this project as important enough to sustain? Does the private sector? What are the resources necessary for implementation? For management? For monitoring? Iniciales __________ Página 1 of 2 ENCUESTA REINSERCIÓN ESCOLAR Y FOMENTO A LA INCORPORACIÓN PRODUCTIVA A JÓVENES DE 13 A 21 AÑOS RESIDENTES EN SECTORES DEL MUNICIPIO DE COTUI, PROVINCIA SÁNCHEZ RAMÍREZ Gracias por compartir su opinión. Es importante para nosotros/as conocer su percepción sobre el proyecto. La información que usted proporciona será únicamente usada para la evaluación final y es anónima y confidencial. 1. Es mujer hombre 2. Edad 3. ¿Participó en el Espacio para Emprender? ¿Cuándo? 4. ¿Qué es lo que aprendiste en el EpE? ¿Lo Mejor? ¿Por qué? 5. ¿Participó en un curso vocacional? ¿Cuál? 6. ¿A que se dedica ahora? a) ¿Si está trabajando ahora, en que trabaja? b) ¿Si ha continuado estudiando, en cual nivel está actualmente? 7. ¿ Cómo se encontró el trabajo? Le ayudó la participación en el EpE en lograr el trabajo? 8. ¿Piensa seguir estudiando? 9. ¿La coordinación del programa fue? excelente bueno regular malo pésimo Iniciales __________ Página 2 of 2 ENCUESTA REINSERCIÓN ESCOLAR Y FOMENTO A LA INCORPORACIÓN PRODUCTIVA A JÓVENES DE 13 A 21 AÑOS RESIDENTES EN SECTORES DEL MUNICIPIO DE COTUI, PROVINCIA SÁNCHEZ RAMÍREZ 10. El proyecto en general le pareció excelente bueno regular malo pésimo 11. ¿Después de terminar el programa se ha reunido con los compañeros? Si su respuesta es si’, ¿con que propósito? 12. ¿Cuál es su meta en su vida? AID517-TO-12-00001 USAID/DR Portfolio Mid-Term Performance Evaluation Annex G:SOW Annex G: Scope of Work Submitted to: USAID/Dominican Republic January 2013 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.