MEGAN GAVIN FOR USAID MID-TERM PERFORMANCE EVALUATION OF THE USAID READ ACTIVITY USAID/DOMINICAN REPUBLIC MONITORING, EVALUATION, AND LEARNING PLATFORM February 20, 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Panagora Group for the USAID/Dominican Republic Monitoring, Evaluation, and Learning Platform Task Order, IDIQ No. AID-OAA-I-15-00025, Task Order No. AID-517-TO-16- 00003. Panagora Group’s Evaluation Team was: Team Leader: Megan Gavin, PhD; Education Specialist: Luis Dominguez, PhD; Evaluation Specialist: Marcelo Castrillo, MD; William Cartier, COP Panagora Group. Recommended citation: USAID Dominican Republic Monitoring, Evaluation, and Learning Platform Task Order. 2017. Final Report. Mid-Term Performance Evaluation of the Read Activity. Kensington, Maryland: Panagora Group. Panagora Group Contact: William Cartier Chief of Party Panagora Group 3209 McComas Avenue Kensington, MD 20895 FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV TABLE OF CONTENTS Executive Summary ........................................................................................................................................................... 2 I. Evaluation Purpose and Questions....................................................................................................................... 8 A. Evaluation Purpose .............................................................................................................................................. 8 B. Evaluation Questions........................................................................................................................................... 8 II. Project Background................................................................................................................................................ 11 III. Evaluation Methods and Limitations .................................................................................................................. 13 IV. Findings and Conclusions................................................................................................................................. 17 A. Question 1........................................................................................................................................................... 17 B. Question 2........................................................................................................................................................... 19 C. Question 3........................................................................................................................................................... 22 D. Question 4........................................................................................................................................................... 25 E. Question 5........................................................................................................................................................... 26 F. Question 6........................................................................................................................................................... 28 G. Question 7........................................................................................................................................................... 30 H. Question 8........................................................................................................................................................... 33 I. Question 9........................................................................................................................................................... 34 J. Question 10......................................................................................................................................................... 37 V. Recommendations.................................................................................................................................................. 38 Annex A. Gender and Social Inclusion ....................................................................................................................... 40 Annex B. Statement of Differences............................................................................................................................. 43 Annex C. Resource List................................................................................................................................................. 45 Annex D. KII List............................................................................................................................................................. 46 Annex E. Fieldwork Schools.......................................................................................................................................... 47 Annex F. Limitations....................................................................................................................................................... 48 Annex G. Data Collection Instruments...................................................................................................................... 49 Annex H. Conflict of Interest Disclosures................................................................................................................ 68 Annex I. Evaluation Scope of Work............................................................................................................................ 71 FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV ACRONYMS APMAE La Asociación de Padres, Madres, Tutores y Amigos de la Escuela CDCS County Development and Cooperation Strategy CETT Centers for Excellence in Teacher Training ESP Effective Schools Program FGD Focus group discussion GoDR Government of the Dominican Republic ICT Information and communications technology INAFOCAM Instituto Nacional de Formación y Capacitación del Magisterio KII Key informant interview MINERD Ministerio de Educación de la República Dominicana PUCCM Pontificia Universidad Católica Madre y Maestra UNIBE Universidad Iberoamericana USAID/DR United States Agency for International Development/Dominican Republic USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 2 EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions Evaluation Purpose This performance evaluation is intended to document lessons learned and to inform the design of future education programs which focus on USAID/Dominican Republic’s (USAID/DR) Country Development and Cooperation Strategy (CDCS) – Goal 1 (literacy). The evaluation provides an independent examination of the overall progress and achievements of the R activity in the Dominican Republic, covering activities from inception to date. The evaluation identifies achievements, performance issues and constraints, and lessons learned from implementation. There are ten evaluation questions (see below). Project Background The USAID Read Program is implemented as a cooperative agreement (AID-517-A-15-00005) by the Universidad Iberoamericana (UNIBE), with the support of World Vision (sub-awardee).1 The Read activity’s period of performance is June 2015 to June 2020. The value of the Read activity cooperative agreement is $20 million. Read supports parents, teachers, and students so that each can play their role in improving early-grade reading. Teachers receive mentoring, training, information and communications technology (ICT), materials and tools to enable them to improve their teaching and to support parents. Parents and caretakers receive materials, information and support to help their children study at home and allow their children to attend supplementary reading programs. Through Read, students have greater access to materials, access to supplementary reading programs, and individualized support to address any disabilities or special needs which may prevent them from learning effectively. Findings and Conclusions Question 1 (What have been the major challenges of the cooperative agreement?). The evaluation team found that there were delays in implementation (nearly one year), especially due to MINERD turnover, and that UNIBE worked closely with MINERD to sensitize the Ministry to the initiative (e.g., location of schools, content focus). Implementation has picked up in the second year of implementation (400 schools to date). INAFOCAM’s new teacher training strategy did not affect the Read program during this period. However, the focus on implementing this new strategy appears to have reduced MINERD interest in Read. 1 Note Lighting Excitement for Excellency in Reading (LEER) was the former name of the project. It was changed to the USAID Read Program and Programa USAID Leer in Spanish. 3 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV Conclusions: Despite delays, thanks to UNIBE’s coordination with MINERD, implementation has progressed and is now on track. MINERD did not show a high degree of interest in Read during its first two years of implementation. Question 2 (How has UNIBE integrated different reading strategies and resources into reading instruction during trainings and hence instruction?). In order to implement strategies, materials (levelled readers, decodable books) are required. Teachers were promised reading materials and were trained on techniques for using them, however there have been delays in approving the reading materials on the part of MINERD. The reading materials were approved in December 2017.2 Teachers concluded that they are using the reading strategies implemented by UNIBE and they are effective. Principals also claim that the reading strategies are effective. MINERD believes they are complementary, in the sense of being useful complements to the Ministry’s own curriculum, but also in the sense of non￾obligatory. The statement of differences (Annex B) presented by one of the evaluation team members qualifies these findings and finds that the reading strategies are not fully understood and utilized by some teachers. Conclusions: The reading strategies are perceived to be effective by various stakeholders. MINERD leadership is less supportive than teachers and Principals. Further research may be needed to understand up-take of strategies by teachers, which may be limited by the absence of teaching materials. Question 3 (Has the training offered by Read helped teachers use effectively the reading strategies incorporated into the new MINERD curriculum?). The evaluation found that teachers are trained in groups by geographic regions/districts and receive up to sixteen hours of training implemented by highly qualified coordinators. The training is considered practical and brings the content of the strategy to the classroom. MINERD acknowledged the importance and quality of the trainings. Mentors, school coordinators, and district technicians also typically participate in the trainings. In the statement of differences (Annex B), however, it was noted that some teachers are not clear on the differences between the MINERD curriculum and the Read methods. Conclusions: A training program exists and is in process throughout the Duarte Corridor. Using high quality trainers and practical exercises makes the training effective, and by having these various actors involved, the training, mentorship and the district/school approach to reading is aligned. Further research may be required to understand why some teachers do not fully understand the Read approach. Question 4 (Has teacher mentoring been effective?). Mentorship is considered the most effective aspect (teachers, MINERD, principals) of UNIBE’s strategy, providing teachers with support to change their practices. Mentorship includes observation, feedback and modeling, and happens seven times a year with the same teacher and mentor. Each mentor has 50 teachers (there are 80 mentors, allowing the 2 As the report was being drafted, the evaluation team was made aware that the materials were approved in December 2017. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 4 program to reach 400 classrooms). The evaluation found that mentorship must be ongoing for it to be effective (not only once or twice). Conclusions: Ongoing mentorship is important from the perspective of all stakeholders in order to make changes in reading instruction. The workload of the mentors is heavy both in terms of the content (i.e., what they do) and stretch of their work (the number of teachers they support). Question 5 (Has school safety improved/ worsened since the Read safe schools intervention?). Gender and social inclusion are not addressed in school culture at large. There is crime and violence in schools (against teachers, between students/bullying), which is largely not addressed, and most problems occur when kids are unsupervised/unengaged in school activities. It is extremely taboo to discuss the LGBTI community in schools. Teachers and schools did not reflect on gender or social inclusion, unless asked, as was the case with key stakeholders. Mentors reflected on the importance of the Safe Schools Program but noted lack of coordination. Conclusions: Gender, social inclusion, and violence prevention training is a crucial aspect of providing safe learning environments; violence and social inclusion are problems, but are not addressed by schools. The Safe Schools Program does not appear to be prominent nor coordinated with the rest of the activity. Question 6 (Are the clubes leyendo [de lectura] offered by UNIBE and World Vision effective?). Parents noted the reading clubs are fun and improve reading for their children, and principals also articulated the importance of the reading clubs. Reading clubs offer two options (to date), lectura and leyendo - they promote levelling support (grade 1 to 3) and the love of reading (grades 4 to 6). Originally the reading clubs did not align with the Read strategy; now, through revisions in programming (new technical staff), the program aligns. Children and volunteers participating in the clubs are visibly motivated and enthusiastic. Conclusions: The reading clubs are well received by parents, community members, students, volunteers, and school staff. There is great demand for the reading clubs to encourage reading abilities and to promote a culture of reading. Question 7 (How successful thus far are Read’s efforts in incorporating MINERD engagement and designing a strategy for long term sustainability?). District technical staff have been engaged for sustainability via trainings, and MINERD directors have participated in trainings (primary first cycle). MINERD directors (of curriculum, evaluation and primary departments) have had ongoing engagement with the Read activity. However, MINERD senior leadership in Santo Domingo is currently focused on implementing a new national teacher training strategy with INAFOCAP. While they say that the Read approach is “complementary” they state that they have no plans at this time to sustain or expand Read’s approach and methods. Conclusions: There is a disconnect between the high level of support for Read at the school and District levels, and the lower level of support at the national level. Read has made gains engaging with MINERD through a horizontal approach to sensitizing MINERD (similar to the way mentors work with teachers in schools). Teachers (who constitute part of MINERD) have been engaged for sustained change in 5 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV classrooms with regard to reading instruction. The reading clubs discussed in question 6 are effective at promoting reading skills and a love of reading. The beneficiaries (parents, students, teachers) are convinced of the approach, this is the key ingredient for sustainability. If they perceive it as effective, then they will sustain the clubs. At the national level, the Ministry’s current focus on the new teacher training strategy (which incorporates some curriculum changes in early grade reading) has resulted in a low level of commitment in the Ministry to supporting and sustaining Read. Question 8 (To what extent has Read generated sufficient knowledge of effective reading instruction and supportive approaches and shared such information with key stakeholders?). UNIBE has produced high quality research that they then shared with the international community. UNIBE has shared less research with MINERD, although recently there was an event where results were presented to MINERD. UNIBE suggested that graphics have been shared with schools on performance (examples exist from the World Bank). Principals report participating in knowledge-sharing events, although they did not receive handouts (to date). Conclusions: UNIBE produced high quality research and has started to share with MINERD, although more is required (for example, the EDUCA congress/forum). UNIBE has produced school-level knowledge, although gaps remain in giving this to the school level for decision-making (see World Bank). Question 9 (How will MINERD’s new teacher training strategy (with Instituto Nacional de Formación y Capacitación del Magisterio [INAFOCAM]) affect the Read Activity going forward?). The evaluation team conducted an interview with MINERD’s Technical-Pedagogical Vice Minister, Denia Burgos and Barbarita Herrera, General Director of Primary Education. In addition, the team conducted various KIIs with other stakeholders involved with teacher training and specifically INAFOCAM. The findings show that MINERD is putting considerable resources into the roll-out of a new national teacher training strategy. The training activities will occur in both Read treatment and control schools at different moments over the next two years. In this sense, they constitute a “co￾intervention bias” which will make it difficult for UNIBE to measure the Read Activity’s impact going forward. Conclusions: The Ministry’s main concern is that Read activities do not interfere with roll-out of the new strategy. They appear to be unwilling to modify the roll out of the strategy to avoid interference with the Read early grade reading interventions. At the same time, the new MINERD teacher training invalidates Read’s impact evaluation design going forward, as it constitutes an insurmountable “co￾intervention bias”. Question 10 (What would the Ministry recommend in order to improve USAID education programming in the Dominican Republic?). The Ministry recommends that USAID support the GODR’s current education strategy. The current strategy consists of a change from competencies to content. The content focuses on reading/writing, mathematics, leadership/ management, and monitoring and evaluation. Senior Ministry officials are critical of the lack of alignment of Read with Ministry policies. They suggest that USAID realign Read interventions with MINERD’s new teacher training strategy going forward. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 6 Conclusions: In the long term, USAID programs are recommended to focus on literacy especially and possibly also management and M&E in the strategy (not mathematics - possibly other donors). In the shorter term, it is recommended that USAID to explore how to realign Read with current Ministry strategy. This could require changes in Read’s scope, to refocus activities towards areas prioritized by MINERD and INAFOCAM. Recommendations 1. Align Read with MINERD priorities. Despite Read’s progress during the first two years, at present MINERD is not committed to sustaining or extending the Read approach. The Ministry is now focused on implementing its own strategy for teacher training, which incorporates some curriculum changes in early grade reading. The Mission should explore with MINERD how to better align Read with the Ministry’s strategy for teacher training in early grade reading. This could require a change in the scope of the cooperative agreement with UNIBE, to focus on activities of interest to the Ministry. One of these areas may be mentoring, in which Read has shown good results. 2. Assess Read’s Impact Evaluation Design. The evaluation team finds that the INAFOCAM teacher training and mentoring activity will likely interfere with the Read activity’s impact evaluation design. The INAFOCAM activity constitutes a major “co-intervention bias” which will impede UNIBE from measuring the independent impacts of the Read interventions. The evaluation team recommends that USAID assess the Read impact evaluation design to confirm the possibility of a co-intervention bias, and if confirmed, suspend this component of the Read activity. In this case, USAID should consider other evaluation activities to support INAFOCAM’s current teacher training strategy. 3. Share findings. Whatever the decisions about the Read activity’s continuity, it should share findings in national forums, especially with MINERD, at the national, district and school levels. This reflects findings and conclusions related to Question #8. It is important that the Read activity share findings in a user-friendly way so that stakeholders fully understand and have ownership of the findings in their schools, which can ultimately support efforts for sustainability in the long term. 4. Continue reading clubs. Again, depending on how the Read activity is realigned with MINERD priorities, it should continue and, if the opportunity exists, expand the reading clubs. Aligned to the Read approach, in considering Question #6, the reading clubs were found to increase involvement, improve skills, and encourage a culture of reading. The clubs are well received in schools and they serve as an optimal space in which parents can get involved in a practical way to help their children learn and love to read. To this end, we recommend increased community resources, for example from the APMAES. 5. Continue the emphasis on mentoring. In realigning the Read activity, USAID should ensure that MINERD can benefit from Read’s success in the mentoring component. Per the systematic qualitative evidence collected related to Question #4, mentoring strengthens school coordinators and district technicians to perform their role and is considered the most effective aspect of the Read program. To implement this recommendation in the second half of the activity with sustainability in mind, the Read activity should begin now to determine who will 7 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV assume mentorship roles in the future, in particular school coordinators and district technical staff with a specialization in Spanish language and reading and bring these individuals on board. 6. Rethink safe schools. The Read activity’s approach to working with schools in the context of crime and violence, especially against women and girls, needs to be rethought in any attempt at realignment with MINERD priorities. Per responses to Question #5, the Safe Schools intervention’s strategy and implementation approach needs to be assessed. The intervention could, for example, have a bigger impact working at the awareness-raising campaign or policy levels, or could include a pilot on gender and social inclusion. The Read activity could look to existing, successful cases considering the context of the Dominican Republic, for example UNICEF’s work in Boca Chica, as a model for Safe Schools. 7. Get it in writing. Based on findings related to multiple questions, the Read activity should get approvals and agreements in writing from MINERD, especially regarding new materials. This is important given the high potential for turnover and the varied power structures within MINERD. Any realignment of Read with MINERD’s new strategy should also be based on written agreements. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 8 I. EVALUATION PURPOSE AND QUESTIONS A. EVALUATION PURPOSE This performance evaluation is intended to document lessons learned and to inform the design of future education programs which focus on USAID/Dominican Republic’s (USAID/DR) Country Development and Cooperation Strategy (CDCS) – Goal 1 (literacy). This evaluation provides an independent examination of the overall progress and achievements of the Read activity in the Dominican Republic. It covers Read activities from inception to date, and addresses sustainability and outcomes. The evaluation identifies achievements, performance issues and constraints related to activity implementation. The evaluation also identifies results and lessons learned from implementation and provides concise, actionable recommendations about which component(s) of Read could be scaled up, modified or re￾designed for the final years of implementation and in future procurements to enhance overall activity performance. The evaluation findings and recommendations will be shared and discussed with USAID/DR, implementing partners, and the Ministerio de Educación de la República Dominicana (MINERD). B. EVALUATION QUESTIONS USAID identified 10 evaluation questions. They are listed below along with their sub-questions, which were designed to clarify the intent of the main question. This evaluation focuses on answering the key questions to the extent possible. The instruments and design of the research endeavor were elaborated based on the proposed questions. Therefore, the findings and conclusions contained in this report are responsive to each of the 10 questions. The recommendations are based on the conclusions and, in some cases, responses overlap between the questions (see section on Recommendations). Q1. IMPLEMENTATION (MANAGEMENT). What have been the major implementation challenges of the cooperative agreement? • Has UNIBE responded effectively to these challenges? • How effective is UNIBE’s coordination with sub-awardees, in particular, World Vision? Q2. READING APPROACHES (TECHNICAL). How has UNIBE integrated different reading strategies and resources during teacher training and hence in classroom instruction? • Has Read implementation (training, resources, materials) been modified in any way as a result of consultations with MINERD? • What are the similarities and differences between Read’s reading interventions and reading interventions in schools where only MINERD is working? Q3. TEACHER TRAINING (TECHNICAL). Has the training offered by Read helped teachers use effectively the new reading strategies incorporated into the MINERD curriculum? • Is the training effective in transmitting the new reading strategies to the teachers? 9 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV • What aspects of the training are most useful? What aspects can be improved? • Have teachers perceived changes in children’s reading skills after applying what they have learned through the Read trainings? What are the main challenges? Q4. TEACHER MENTORING (TECHNICAL). Has the teacher mentoring been effective? • How frequent does the mentoring occur? Is the frequency appropriate? • How is the Mentor selection process conducted? Are the Mentors qualified? • Is the mentoring process effective in improving the quality of reading instruction? If so, how? • What can be done to improve teacher mentoring? Q5. SAFE SCHOOLS (TECHNICAL). Has school safety worsened/improved after the Read safe schools intervention? • How is school safety defined and operationalized by UNIBE? • How does UNIBE assess whether the safe schools and violence prevention interventions are effective? • How does UNIBE measure the impact on students of the safe schools and violence prevention intervention? Is there a minimum threshold that reflects “success”? • How effective do teachers consider the gender, inclusion and safe schools training? Has it been useful? • Has the safe schools program mentoring follow up been useful? How? Is any change needed? Q6. COMMUNITY INVOLVEMENT (TECHNICAL). Are the Clubes Leyendo [and de Lectura] supported by UNIBE and World Vision effective? • What is the process followed for students to participate in the Clubes? Is the frequency of club meetings appropriate? Is the time of day appropriate? • Are parents and the community involved in their children’s learning? Has their involvement increased through the Clubes? If so, in what way? • Has Read been effective in creating and strengthening the Asociaciones de Padres, Madres, Tutores, y Amigos de la Escuela (APMAEs)? • Have parents of students participating in the Clubes seen any improvement in their children’s reading/ writing? In what aspects? USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 10 Q7. ENGAGEMENT WITH MINERD (SUSTAINABILITY). How successful thus far are Read’s efforts in incorporating significant MINERD engagement and designing a strategy for long-term activity sustainability? • Are the mechanisms for engagement with MINERD effective? • How does UNIBE present the Read activity’s reading success to MINERD? • What is MINERD’s opinion about the effectiveness of this activity to date? Q8. KNOWLEDGE MANAGEMENT. To what extent has Read generated knowledge of effective reading instruction and supportive approaches and shared such information with key stakeholders? • Has UNIBE developed a comprehensive analysis plan to identify all of the topics and methods of analysis of Read research data? • Has UNIBE made full use of the data collected in the baseline? • Do Read’s research reports explain clearly the questions, methodology, limitations and findings from statistical analysis? • In reporting on research results, does UNIBE explain clearly its methodology (the use of randomized control trials) to measure impact? • Are Read’s research products appropriate for dissemination and discussions with key stakeholders? Q9. RECENT CHANGES IN READ CONTEXT. How will MINERD’s new teacher training strategy (with Instituto Nacional de Formación y Capacitación del Magisterio [INAFOCAM]) affect the Read Activity going forward? • What challenges will the Read activity face while working in parallel with another institutions within the same schools? • What will be the impact of the new training strategy on the Read control group? Q10 MINERD RECOMMENDATION. What would the ministry recommend in order to improve USAID education programming in the Dominican Republic? GENDER AND SOCIAL INCLUSION QUESTIONS The following additional sub-questions were added to the analysis (after discussion during the in brief with USAID). These questions help to guide the gender and social inclusion analysis of the report by pulling out and explicitly asking questions related to gender and social inclusion. More detail on the analysis of gender and social inclusion is provided in the methodology section, below. See Annex A, Gender and Social Inclusion, for findings and conclusions. • Q2 (reading strategies, resources and materials) 11 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV – Is a gender lens applied to the materials created by UNIBE (and/or other socially￾excluded groups)? How? • Q3 (teacher training) – Does teacher training apply a gender lens? – Do teachers have different perspectives on girls’ and boys’ (other socially-excluded groups) performance and achievement? • Q5 (safe schools) – How effective is gender and social inclusion promoted in safe schools? • Q6 (parent/ community participation) – How does participation vary by gender- mothers/ fathers, grandmothers/ grandfathers (and other socially-excluded groups), etc.? II. PROJECT BACKGROUND The USAID Read activity is implemented as a cooperative agreement (AID-517-A-15-00005) by the Universidad Iberoamericana (UNIBE) (prime awardee), with the support of World Vision (sub-awardee). The Read activity’s period of performance is June 2015 to June 2020. The value of the Read activity cooperative agreement is $20 million. The Read activity employs USAID global best practice and experience in the Dominican Republic around engaging communities together with schools to improve early-grade reading. Read supports parents, teachers, and students so that each can play their role in improving early-grade reading and some numeracy. Teachers receive mentoring, training, information and communications technology (ICT), materials and tools to enable them to improve their teaching and to support parents. Parents and caretakers receive materials, information and support to help their children study at home and allow their children to attend supplementary reading programs. Students have greater access to materials, access to supplementary reading programs, and individualized support to address any disabilities or special needs which may prevent them from learning effectively. In addition, Read creates an enabling environment within the community through campaigns promoting literacy, safe schools and education for all. These are coupled with incentives that mobilize wider community contributions in the form of youth volunteers, including from groups supported by another USAID/DR activity, the USAID-funded Alerta Joven activity, and the wider community. Campaigns raise awareness in the communities and engage parents to advocate for greater equality in learning for boys and girls. The campaigns employ information generated through community scorecards that provide information on the extent to which the quality of education is improving. It is important to understand that the Read activity, while working to improve early-grade reading skills, covers important additional areas, including inclusion and inclusive learning environments. In addition, Read covers gender, especially gender equity for program beneficiaries (children, teachers and USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 12 caregivers). Lastly, Read covers sustainability and uptake by the government. These aspects of the theory of change and developmental hypothesis are all closely-related and the evaluation analyzes their interconnectedness. 13 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV III. EVALUATION METHODS AND LIMITATIONS The mid-term performance evaluation of the Read activity took place between July 14 and November 30, 2017, with fieldwork in the Dominican Republic October 29 to November 15, 2017. This section describes the methodology employed in conducting the performance evaluation. In order to ensure the validity and reliability of the study, various methods and sources were consulted. First, for each question, the evaluation team started by reviewing relevant activity documentation. With this base, the team then gathered data from additional sources (e.g., interviews or discussions with parents followed by direct observation in classrooms). In this way the team could confirm: what UNIBE reported it was doing, what parents/teachers/principals said UNIBE was doing, and, in turn, what we, in fact, observed in classrooms. DOCUMENT REVIEW During the initial phase of the evaluation, the team conducted a thorough review of background documents related to Read and its implementing environment. These documents provided the team a deeper understanding of the activity’s operations to date. This helped the team finalize the evaluation design and data collection tools, and informed the overall activity assessment process, including developing conclusions and recommendations for future programming. The list of documents reviewed is provided in Annex B, Resource List. KEY INFORMANT INTERVIEWS The evaluation team conducted 20 key informant interviews (KIIs) with MINERD, INAFOCAM, local and international NGOs (Dream), donors (the World Bank, USAID, UNICEF), UNIBE and partners (World Vision, Fe y Alegria), and private sector partners (for example Fundación Inicia). In total, the team interviewed 31 individuals. Data collection protocols that addressed the evaluation questions guided KIIs, but the team conducted semi-structured interviews (identifying probes to follow up on information related to key questions). The semi-structured approach allowed flexibility in the discussion and a natural flow to the conversation when the evaluation team identified a point of interest to the research questions. The evaluation team conducted field research in 24 schools supported by the Read activity. The team interviewed school principals and one teacher per school. Data were recorded electronically. A list of the KIIs conducted is provided in Annex C, KII List. GUIDED DISCUSSIONS A total of 10 guided discussions using ethnographic techniques were conducted with parents. Originally the intent was to hold focus group discussions (FGDs) with parents, but this proved to be difficult. Often parents would not show up or the school principal/director resisted calling the parents. Therefore, in some cases the evaluation team solicited conversations with parents and groups of parents (1 or 2 parents) while outside of the classroom/school building. This lead to interesting findings, especially because the conversations were not anticipated. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 14 FIELDWORK Fieldwork consisted of arriving at a school, conducting an interview with the director, conducting a classroom observation, then conducting an interview with the same teacher observed. Then, in 10 cases, the evaluation team conducted a guided conversation with parents. A total of 24 schools were visited (please see Annex D, Fieldwork Schools). The team selected schools from the following Regional Education Bureaus: Regional 04 San Cristobal, Regional 06 La Vega, Regional 08 Santiago, Regional 09 Mao, Regional 10 Santo Domingo, Regional 15 Puerto Plata, Regional 15 Santo Domingo and Regional 16 Cotui. The team stratified school selection by Regional Education Bureau and Year of Entry into the program (i.e., Year 1 or Year 2). Selection was proportional to the number of schools within each of the Regional Education Bureaus. The team achieved a representative sample of rural, urban and peri-urban schools. CLASSROOM OBSERVATIONS The evaluation team undertook one classroom observation per school to collect qualitative data in order to gain a deeper understanding of the implementing environment and how the curricula are delivered. The evaluation team worked in direct collaboration with the site administrators and facilitators to identify appropriate opportunities for this observation and sought to minimize disruption of the normal classroom environment as much as possible. Observers used a rigorous classroom observation tool that reflected what is expected to be seen in the Read approach to guide observations. Observations did not exceed 45 minutes. Each observer recorded the observed behaviors in the observation tool during the observation. Then, after checking the tool, the team member submitted the observation tools daily to facilitate rapid analysis of observation results. Data were recorded electronically. DATA ANALYSIS Dedoose was used to code and analyze qualitative data gathered through the KIIs using systematic qualitative methods. The team leader developed a coding system to identify and respond to information sought in the key evaluation questions and oriented each team member to the coding process to increase inter-rater reliability. In addition, the team coded for demographic information, such as gender, age, ethnic background, economic and educational status, etc., which the team also considered during analysis. While the team collected data from a representative sample in these regards, the sample was not large enough to report meaningful disaggregated information in this report. The team anonymized qualitative data in the evaluation report in order to ensure respondent confidentiality; only the respondent’s data collection location is identified with any specific quotations, to provide context for the quotation. Qualitative data were enumerated but not quantified. In other words, The team coded the data and then counted the frequency of codes. Illustrative quotes were pulled from Dedoose and the interviews to support and give depth to findings. Similarly, the team treated classroom observation categorical data in a “quantitative way.” The team used an electronic open source data collection software for the classroom observations. Using the 15 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV software Commcare, the team downloaded the observation data in Excel and generated the quantitative tables/ figures contained within this report. The team does not make statistical inferences or assumptions on causation given the small size of the sample (n = 24) and the design of the study. Nevertheless, the visual representations are accurate in that the sample of observations is reflective of the overall composition of the Read activity. All data have been provided and are easily accessible to the USAID/DR MEL Platform team to share with USAID (per USAID evaluation policy). GENDER AND SOCIAL INCLUSION METHODS Consistent with USAID’s evaluation policy and recognizing that effects of integration and the success of the activity might vary across gender; the team applied a gender perspective to the entire evaluation process. Starting with the background document review, data related to gender-based variances in outcomes were examined, and activity documents were reviewed with a gender-sensitive lens, to inform a better understanding of gender dynamics in the implementation environment. The evaluation team included both female and male researchers, and a gender balance among respondents was sought during the recruitment phase. While it was not possible to achieve a gender balance in teacher respondents, which were all women, this is representative of the gender balance in teachers at the relevant grade levels. During the data analysis phase, all participant-level data (both qualitative and quantitative) were disaggregated by sex. This analysis of gender aspects of the Read intervention and discussion of findings, and, where appropriate, conclusions, are included in the evaluation report. Ethical considerations were considered throughout the evaluation, from inception phase, design, implementation, data collection, analysis and reporting. In other words, especially during the data collection process, we: protected the anonymity of participants, asked for permission prior to conducting an interview, and did not ask any questions that could be unsettling to the participant. LIMITATIONS There were a series of limitations that were mitigated by the evaluation team. A full list and description of each constraint can be found in Annex E, Limitations. Here a brief summary is provided. First, all targeted KIIs were conducted with the exception of one. The one not conducted was with the Director of Planning from MINERD. The evaluation team spoke with all other directors: curriculum, primary, primary cycle, and assessment and evaluation. However, speaking with the Director of Planning was important to understand prospects of sustainability. The Director was out of the country during the field work phase of the evaluation. The team then tried to meet with him during the report writing phase, but he was not available to meet. The dates for this evaluation were originally scheduled for the end of September/ beginning of October. Unfortunately, due to poor weather conditions (including hurricanes), the timeline was delayed. Furthermore, the date of the evaluation was revised because of the ever-evolving state within MINERD; USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 16 at one point, the research questions had to be entirely overhauled in order to align with the new context (see especially questions 9 and 10). Lastly, MINERD did not grant authorization to conduct research in schools until approximately 5 days before fieldwork began. Coordination was a limitation in the sense that in some performance evaluations there is more support from the implementing partner. On the one hand, such support must be managed carefully so as not to interfere with the objectivity of the study. On the other hand, coordination with UNIBE, especially making contact with schools, coordinating visits, and coordinating parent discussions -- would have increased efficiency. Nevertheless, the team on its own was able to achieve this. While it was not possible to make contact with the implementing partner sooner (as mentioned, the team only received authorization from MINERD five days prior to fieldwork), in future evaluations it would be preferable to make contact with the implementing partner earlier in the process. Timing was also compromised due to holidays in the United States and the Dominican Republic (November 6 in the Dominican Republic and Thanksgiving in the United States). The implications are to have strong internal communication and controls to ensure sufficient review of deliverables, etc. 17 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV IV. FINDINGS AND CONCLUSIONS This section presents the evidence-based findings for each research question. The evidence includes data from classroom observations, interviews and guided discussions, as well as background literature. Conclusions are drawn based on the findings. Section V presents actionable recommendations based on these conclusions. A. QUESTION 1 What have been the major implementation challenges of the agreement? At the time Read was in the design phase, MINERD, through the Instituto Nacional de Formación y Capacitación del Magisterio (INAFOCAM), had finalized planning for a new national teacher training strategy, which was intended to be implemented by universities. According to INAFOCAM’s policy documents, this strategy would include new training modules on early grade reading.3 Implementation of the new strategy, while initiated in late 2014, was delayed until 2017. Thus, during the first two years of the Read activity, the Ministry’s new strategy did not interfere with Read implementation. However, as explained in interviews with senior Ministry officials, the focus on the design, planning and roll out of the new strategy meant that the Ministry was “less interested in pilot programs” carried out by donor agencies and other organizations. Initially, Ministry officials were not engaged with Read. Constant effort by UNIBE resulted in growing engagement and approval of Read school selection and training. Approval of Read materials was delayed considerably. MINERD officials 3 Dominican Republic. Ministry of Education. Marco de formación continua: una perspectiva articuladora para una escuela de calidad. – Santo Domingo: Inafocam, 2013. QUESTION 1 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS There were delays in implementation (nearly one year), especially due to MINERD turnover, but now on track MINERD showed a low level of interest in the Read activity in the first two years. Despite delays in Ministry approvals, thanks to UNIBE’s consistent efforts to engage with MINERD, implementation has progressed and is now on track. While MINERD was developing a new strategy for teacher training that included modules on early grade reading, this did not impact Read implementation in its first two years. Prior to the start of Read, in late 2014 MINERD initiated a national teacher training strategy, including changes to teaching of early grade reading. MINERD’s new teacher training strategy was delayed in its planning and procurement in 2015-2017. UNIBE worked closely with MINERD to sensitize the Ministry to the initiative (e.g., location of schools, content focus) Implementation has picked up in the second half of implementation (400 schools to date) USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 18 said that the Read activity was not included in the transition briefings between the outgoing and incoming Ministry leadership with the change in government in 2016. The issues related to the impact of MINERD’s new teacher training strategy on the Read activity will be discussed in more detail in the responses to Questions 7 and 9 below. According to KIIs with Read stakeholders, initially it was not clear how UNIBE would work with MINERD. Then MINERD and UNIBE sat at the same table and UNIBE explained what, where and how UNIBE intended to implement the activity. Now, MINERD (Dir. of Primary) is content that Read shares a common goal with the Ministry to teach reading in the early grades. See KII reflections below. “Before it was not clear how we would work together (UNIBE and MINERD). Now we are content with the project. We like the most recent versions of the materials. We are happy with the coordination.” -KII 001 MINERD Dir. of Primary MINERD notes that Read is using a more traditional methodology. While Read focuses on letters, sounds and words, MINERD’s methodology focuses on the context and communication. Nevertheless, MINERD is pleased with the materials and the coordination. Furthermore, MINERD acknowledges that their teachers appreciate Read’s methods. Figure: Implementation Perspectives Despite the bumpy start, from the point of view of key informants, 66.6 percent believe that implementation has been effective (see graphic below). Now, the project is on track to achieve its goal of reaching 400 schools. In large measure, this is due to UNIBE’s efforts to work alongside MINERD, and to convince MINERD to allow Read’s approach to reading to be implemented as a “complementary” method in the 400 schools alongside of MINERD’s own curriculum (further discussed below). However, in the first two years, UNIBE was unable to obtain approval for distribution of the Read materials. 19 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV B. QUESTION 2 How has UNIBE integrated different reading strategies and resources? QUESTION 2 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS In order to implement strategies, materials (levelled readers, decodable books) are required Without materials, it will be difficult to implement Read strategies in 2018 [Note as of December 2017 there are updates in the relationship with MINERD and the materials] Without written approval from MINERD, the state of materials is uncertain The reading strategies are effective from the point of view of various stakeholders. MINERD senior officials say that the Read methods are complementary to Ministry curriculum, only in the 400 schools. The evaluation field work suggests that in some schools there is still work to be done on teachers’ understanding and use of Read approach and methods. Teachers were promised reading materials and were trained on techniques for using them There have been delays in approving the reading materials on the part of MINERD. This may have hindered their full use. It is anticipated that reading materials will be approved in 2018, but this is not official Many teachers say that they are using the strategies and they work. Probing in some schools shows difficulties in understanding the Read approach and methods. Principals claim that the reading strategies are effective MINERD confirms that the reading strategies are complementary. This means that they are allowed in the Read schools, but they will not be incorporated into the curriculum for now. According to KIIs with stakeholders, Read implements a very precise methodology to teach reading. In trainings, Read breaks down the approach to learning how to read. Read uses methodologies such as read-aloud and make teaching-learning, which focus on teaching how to read through practical examples. This is different than what teachers are usually exposed to in in-service and pre-service training, where the approach is largely theoretical. See reflections below. “UNIBE is very precise. They know how to break down teaching reading. This helps to make the teaching practical. This is different because most teacher training is theoretical.” -KII 002 DR Education NGO “Before the materials were not good. Together we revised them. Now they are much better.” -KII 003 MINERD (Dir. of First Cycle) Similarly, KIIs with MINERD reflected on the process of revising the reading materials (i.e., the leveled readers and the decodable books). At first, MINERD was not satisfied with the materials. Now, MINERD believes that the materials are much improved. From the point of view of MINERD, the materials were revised to better reflect the context of the country and the curriculum. The interview with senior MINERD officials found that the Ministry saw the Read methods and materials as USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 20 “complementary” to the official curriculum. By this, they explained that Read methods were permitted in the treatment schools but not incorporated into the curriculum. As pointed out by the statement of differences in Annex B, not all teachers fully understood the new approach and how it differed from the existing curriculum. Classroom observation data is useful to get a snapshot of the integration of reading practices in the classroom, especially regarding the materials available. As seen below, while 41.7 percent of the classrooms had a textbook for each child, an equivalent number had no textbooks per child. Figure: Textbooks Available Figure: Using Textbooks Furthermore, in cases where there were textbooks, only 4.2 percent of classes were using the books in their lesson. While not a representative sample this statistic merits further exploration. For example, does it reflect that the textbooks provided by MINERD are not relevant or useful, or possibly out of date? Does it reflect that teachers are not adequately trained on how to integrate the textbook into their teaching? Or is it related to the fact that at least 40 percent did not appear to have textbooks? The project at the very least could work with the textbook in its trainings to demonstrate to teachers how to integrate the text into lesson planning and delivery of instruction. This can be monitored by the project. Similarly, 75 percent of classrooms had additional reading materials (for example story books, posters, newspapers, cards with letters or words, dictionaries, etc.). A literature-rich environment is important for learning to read and write. In other words, children can only learn to read if they have things to read. On the other hand, only 12.5 percent of classes were using the additional reading materials during the lesson. 21 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV Figure: Other Materials Figure: Using Other Materials In addition to the other materials referenced above, the evaluation team looked at reading materials on the walls. As shown in the figure below, and similar to findings of other materials, 75 percent of classrooms had reading materials on the walls. Figure: Materials on Walls In the statement of differences presented below, in Annex B, one of the evaluation team members presents different findings on the effectiveness of UNIBE’s integration of different reading strategies. He concludes that teachers in Read schools have some way to go in incorporating the strategies imparted by Read. In probing questions, most of the teachers and Director interviewed in ten schools could not explain the differences between the Read methods and the MINERD approach to teaching of early grade reading. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 22 C. QUESTION 3 Has the training offered by Read helped teachers integrate the reading strategies effectively? According to our KIIs, MINERD’s curriculum is in a transition phase, and remains very theoretical. MINERD is building knowledge and experience and Read complements MINERD’s strategy with practical exercises. For example, if MINERD promotes the “communication focus” (enfoque comunicativo, in Spanish) using for example the “my own name” approach (mi nombre propio, in Spanish); it builds off of the child’s name. This is a method promoted in the Dominican Republic by USAID under the Centers for Excellence in Teacher Training; a regional initiative in the early 2000s. Then Read complements this by using a method to teach letters and sounds (i.e., phonics). For more information on the differences between a whole language and a phonics￾based approach, see the textbox to the left. Another challenge is the inconsistencies between the perceptions of the approach from the central and the local levels. Teachers and parents went to school and learned a traditional approach (phonics) to reading. They draw from that traditional approach (more phonics-based, similar to Read) when supporting children’s reading in classrooms and at home. Given this reality (especially in resource-depleted environments) it is unlikely that teachers can implement the whole language constructivist approach MINERD promotes, which is distinctly QUESTION 3 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS Teachers are trained in groups by geographic regions/ districts A training program exists and is in process throughout the Duarte Corridor Using high quality trainers and practical exercises makes the training effective By having these actors involved, the training, mentorship and the district/school approach to reading is aligned Teachers receive up to sixteen hours of training Training is implemented by highly qualified coordinators Training is practical; brings the content of the strategy to the classroom MINERD acknowledges the importance and quality of the trainings Mentors participate in the teacher trainings School coordinators and district technicians participate Whole Language vs. Phonics-> Balanced Approach (Succeed to Read, 2017) Phonics: Children are taught how to sound out new words via consonant letter sounds, blended sounds, short vowels, etc. (a part >whole method) Whole language: Teachers use connected print to introduce reading, words as whole units, analyzing words in context for meaning (whole >part method) Research: Whole language leads to writing early (a MINERD priority); on the other hand, readers tend to use phonics to decipher new words Balanced approach: Use auditory preparation at home; phonics and whole language instruction in school 23 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV different from the traditional approach. Students need skills – and the Read traditional, phonics-based approach builds the capacity for teachers to teach and students to learn these skills. It is important to note that UNIBE has incorporated reading strategies contemplated in the new curriculum, such as phonics. On the part of MINERD, they confirm that senior leadership (i.e., Directors) have participated in trainings on the Read approach and that the themes are aligned to the curriculum. More recently (i.e., in the second half of the project), technical staff at the regional level and pedagogical coordinators from schools have been more involved. They confirm that the trainings are practical and give the teachers the opportunity to practice and role play. For example, the trainings include various activities such as the community store or reading with newspapers. These practical exercises help the teacher, and the practical training they receive is further reinforced during the mentorship discussed below. “In the teacher trainings UNIBE presents different themes aligned to the curriculum. The teachers are able to practice implementing the curriculum. UNIBE gives materials and facilitates.” -KII 004 MINERD (Dir. of Primary) The first step in implementing a systemic approach to teaching reading is for the teacher to have a lesson plan. Here the team reviewed the statistics from our classroom observations associated with lesson plans. It is important to keep students engaged in reading through a variety of methodologies, such as reading aloud, engaging in a dialogue, using different materials, etc. Furthermore, these different techniques build different skills associated with reading and serve different student learning needs (i.e., multiple intelligences as promoted by Harvard Professor and learning expert, Howard Gardner). Figure: Lesson Plan Used by Teacher As seen in the figure above, 83.3 percent of teachers appeared to have a lesson plan, as observed during classroom observations. Furthermore, teachers employed a variety of methods to teach literacy. All teachers observed engaged students in a dialogue (100 percent). More than 90 percent (91.7%) read USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 24 aloud. Nearly 80 percent write on the board. The fact that teachers use techniques such as writing on the board and probing and reading aloud suggests that they do not have sufficient materials to employ techniques which require materials, such as those developed and trained on by Read. Only in 41.7 percent of classrooms did teachers demonstrate teaching (either physically or with materials). Students also engaged in a variety of ways. All students (100 percent) engaged in oral reading, while only 4.2 percent engaged in silent reading. Again, this reflects that students are likely reading from a shared source such as the blackboard, not with their own materials. Students were given the opportunity to practice (58.3 percent) but fewer applied what they were learning (37.5 percent). Good reading practices allow children a variety of opportunities to demonstrate what they have learned. Figure: Student Engagement in Learning Figure: Teacher Methods Employed 25 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV D. QUESTION 4 Has teacher mentoring been effective? According to the KIIs, mentorship is extremely valuable in changing teacher practices. Stakeholders note that while training is beneficial in the moment, when teachers return to their schools and classrooms, it is difficult to implement the new strategies they have learned. Mentorship provides them with ongoing support. The people who serve as UNIBE mentors participate in the trainings, and therefore align what they do in the classrooms with the trainings. See reflections below. “The value of mentoring is that it helps change patterns, while trainings are good in the moment then you try to implement in the classroom and it is very challenging because you do not have feedback.” -KII 005 DR Education NGO There is more mentoring from the project. The project gives more than the district. The mentor came 5 times this year. It is more intensive. The technical staff haven’t come to support reading. KII 002 Teacher The UNIBE mentors are high-quality, well-trained professionals. UNIBE invested in onboarding them to prepare them. The KIIs confirm it is difficult to find this level of expertise in reading within the MINERD structure, including with technical staff in the districts. Nevertheless, in the future, either these technical staff or school pedagogical coordinators would have to assume the mentoring role. According to QUESTION 4 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS Mentorship is considered the most effective aspect (teachers, MINERD, principals) Ongoing mentorship is important from the perspective of all stakeholders in order to make changes in reading instruction The workload of the mentors is heavy both in terms of the content (i.e., what they do) and stretch of their work (the number of teachers they support) Mentorship provides teachers with support to change their practice Mentorship must be ongoing for it to be effective (not only once or twice) Mentorship includes observation, feedback and modeling Mentorship happens seven times a year with the same teacher/mentor Mentors have 50 teachers each (there are 80 mentors, allowing the program to reach 400 classrooms) USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 26 MINERD, it would be useful to have more data and information on the mentoring approach (more discussion is given towards knowledge management in question 9 below). “LEER has a strong approach to accompanying teachers. We believe it is successful. We need to do more to follow-up and evaluate what they are doing.” -KII 006 MINERD (Dir. of Curriculum) “There is a mentor who helps me. He gives ideas when I have challenges. He tells me how to do things. He provides pedagogical strategies that help - for example, the lectura coral, and this was something he shared with me. He said that no matter what content - I should start with reading in any subject.” -KII 001 Teacher “There is a mentor who visits me every month. They observe me in the class. She also helps me with the phonetic awareness…works directly with children to read. She has helped me a lot with a strategy - the lectura/escrita en un minute - how many words they write in a minute.” -KII 003 Teacher E. QUESTION 5 Has school safety improved/worsened after the Read safe schools intervention? It is difficult to attribute the state of school safety to Read’s Safe Schools intervention without a treatment control design. The evaluation team reflected on the state of school safety at large, findings QUESTION 5 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS Gender and social inclusion are not addressed in school culture at large Gender, social inclusion and violence prevention training is a crucial aspect of providing safe learning environments; violence and social inclusion are problems, but are not addressed. The Safe Schools Program does not appear to be prominent nor coordinated with the rest of the activity. There is crime and violence in schools (against teachers, between students/bullying); violence is not addressed in schools; most problems occur when kids are unsupervised/unengaged in school activities It is extremely taboo to discuss the LGBTI community in schools Teachers/ schools did not reflect on gender or social inclusion, unless asked Only when probed, did key stakeholders discuss gender and social inclusion Mentors reflected on the importance of the Safe Schools Program, but noted lack of coordination 27 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV from the field, and reflections from KIIs. It is known that the Safe Schools intervention was modified before the final materials were developed. According to the M&E Specialist of UNIBE, “all materials reflect the current approach being implemented”. According to the most recent quarterly report (Q4 2017), school visits to address gender and social inclusion by Read started in mid-April. Mentors conducted one visit to each school, where they held a meeting with the school director, psychologist and representative from the teaching staff. The meeting focused on follow up of agreements and tasks from the training sessions, and the school’s situation in terms of gender, inclusion and violence. An observation instrument was employed. In addition, upon consent, data were gathered from sixth-grade students on school climate, gender, school violence and inclusion through classroom observations with a standardized observation instrument (results are pending Q1 2018). According to the documentation, 16,107 students are benefiting from the Safe Schools intervention. However, there was little to no evidence of the Safe Schools intervention during field visits for this evaluation. According to KIIs, the primary challenge with regard to school safety is with bullying in school. There are many over-age students - for example, a 14-year-old student in the third grade – which presents an imbalance in power/strength that lends itself to bullying. It is very difficult for a teacher to counter bullying/ violence in that situation. Furthermore, most of the time, bullying happens when students are not supervised (recess or before/after school) and unengaged. This suggests that there is an opportunity for Read to engage these students in productive activities, such as those associated with reading. Currently, there is not a school culture against violence. “If there is no supervision. Also they are not engaged. There is no class culture against violence. There is no consequence. The teachers do not have the skills or abilities to handle the violence. There is no conversation about it in the schools.” -KII 007 DR Education NGO In addition, there tends to be a relationship between gender and violence, such that boys are more likely to be bullies. There are stark rates of femicide, where women are the primary targets of homicide, in the Dominican Republic. In addition, there is a relationship between boys and learning disabilities. Therefore, there is a need to look more closely at learning needs, particularly of boys, and whether their engagement or lack of engagement in school (as a result of learning difficulties) influences discipline problems and, in turn, higher rates of violence in school against students and teachers (both in terms of social/emotional and physical bullying). One example is provided by UNICEF, which works directly with MINERD against violence in Boca Chica. According to a KII with UNICEF, the teachers are desperate for support against violence in schools. They do not know what to do in the school given the context of violence that exists in the home and community. The program focuses on norms for working and living together in the school community, and how to help students change behavior rather than removing them from the school (which is often the form of action taken by administrators in the face of school violence). The purpose of the pilot is to give schools and administrators tools and guidance to make this change at the school USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 28 level. While there was initial resistance, now teachers and administrators are grateful for the project. It currently operates in 32 primary and secondary schools. F. QUESTION 6 Are the clubes leyendo y de lectura supported by World Vision effective? Across the literature, interviews, and data collected from fieldwork, it is evident that the participation of parents and caregivers, as well as the community at large, is an important aspect in improving reading outcomes. Research supports the claim, especially more recent studies such as TERCE, which notes the relationship between associated factors, such as parental involvement, and student performance. In the Dominican Republic and in the Read activity, one aspect in which there is parental/caregiver involvement is through the Asociaciónes de Padres, Madres, Tutores y Amigos de la Escuela (APMAEs). These are school organization structures, as defined in the “Ten Year Education Plan”, political strategy number 8, to increase participation of parents and family. Specifically, roles are assigned to membership in the APMAE, for example a president, secretary, etc. However, there is still a lack of knowledge about the APMAE in terms of how to translate those positions into actual roles and actions to promote the education of members’ children, including to support reading. The APMAEs need to be trained as well as motivated. As part of its agreement with UNIBE and Read, World Vision conducted a baseline to assess the capacity needs of the APMAEs. Whereas before school management was highly centralized, under decentralization, APMAEs now have some defined responsibilities. The APMAEs have responsibilities such as small purchases and their own bank accounts. Furthermore, each school class has three parents that represent each grade as part of the APMAE structure. World Vison strengthens the capacity of the APMAEs especially with regard to pedagogical aspects. World Vision helped strengthen APMAEs’ use of QUESTION 6 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS Parents noted the reading clubs are fun and improve reading for their children The reading clubs are well received by parents, community members, students, volunteers and school staff. There is great demand for the reading clubs to encourage reading abilities and to promote a culture of reading. Principals also articulate the importance of the reading clubs The reading clubs offer two options (to date) lectura and leyendo- they promote levelling support (grade 1-3) and the love of reading (grades 4-6) Originally the reading clubs did not align with the Read strategy Now, through revisions in programing (new technical staff), the program aligns Children and volunteers participating in the clubs are visibly motivated and enthusiastic 29 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV information for decision making. (Note: World Vision also works with other donors such as the World Bank in this. See below.) In addition to APMAE organizational capacity strengthening, and of utmost importance to the Read activity, is World Vision’s work with reading clubs - both in terms of leveling children and creating a love for and culture of reading. Often children in grades 1 to 3 move through the system and pass through to fourth grade with reading deficits. Teachers use a diagnostic to identify children with such reading deficits. These children participate in the “Leyendo” program. World Vision works with learners in the Leyendo program for 15 weeks inside and outside of the school and in the extended day (afternoon) program. In total, there are 7,000 children in the program. Specifically, each club has about two volunteers and 18 children and youth. The program includes games, work with a box of letters, and other dynamic and fun ways to reinforce and strengthen reading of letters, words etc. (aligned to Read and the national curriculum). The idea is that the experience is fun, not a punishment. World Vision works with the school director to find volunteers who have the right profile (for example, a degree in psychology or pedagogy) to support these children. World Vision has changed its staffing to ensure that its work aligns with UNIBE’s and MINERD’s approach. The Leyendo program includes aspects such as the alert call. This notifies the team of professionals when a child is not advancing. Currently there are about 200 children that have not advanced. The lead tutor is notified and will help more with these specific children. Photo: Reading Club/ “Club de Lectura” USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 30 Less visible (and outside the original focus of this evaluation question) are the reading or “Lectura” clubs. The purpose of these clubs is to promote reading for children in grades 4 to 6 (please see the photo of a reading club). Again, the methodology is linked to MINERD’s curriculum, with World Vision promoting a more systematic approach linked to Read. The children meet with their volunteers and have learning experiences directed by the volunteers and by “student leaders of reading.” They use a “Passport of Reading,” which is a colorful reading material approved of by MINERD. In this passport, the children document the books they are reading. The children feel proud of what they have read, which motivates their reading. Finally, there will also be a summer club for children in first grade who have never been to school to build a foundation for reading. From the point of view of participants (and observation) the children (and volunteers) who have the opportunity to participate in these clubs are motivated and interested in reading. Future research (i.e., UNIBE’s randomized controlled trial) will indicate the effect of the clubs on reading performance. Nevertheless, the first order effect of diagnosing students in need of support and building a love for reading should not be minimized. Furthermore, the clubs provide a space for parents to organize around a shared problem and be involved. G. QUESTION 7 How successful are Read’s efforts in incorporating MINERD engagement for sustainability? Sustainability is defined through two dimensions: first, through the prospects of ownership and change at the local level by district technicians and teachers; and second, ownership by central leadership of MINERD. According to KIIs, the teacher is the point at which interventions can accomplish sustainable change. Once the teacher changes his/her practices and witnesses students learning, the teacher will continue QUESTION 7 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS District technical staff have been invited to trainings Read has made gains engaging with MINERD through a horizontal approach to sensitizing teachers and directors (like the way mentors work with teachers in schools) The support from school and District level stakeholders contrasts with low support from MINERD leadership of for the Read approach and teaching methods. Ministry officials in Santo Domingo do not support adoption of the Read approach. They describe it as complementary to the official curriculum in the 400 Read schools. MINERD sees the Read activity as being in potential conflict with the roll out of the INAFOCAM teacher training strategy. MINERD directors have participated in trainings (primary first cycle) MINERD directors (of curriculum, evaluation and primary departments) have had ongoing engagement with the Read activity In the design, award and startup phases of the Read activity, USAID and UNIBE engaged with MINERD but not INAFOCAM. MINERD senior officials say that Read is not aligned with current Ministry and INAFOCAM strategy for teacher training, which includes changes to teaching of early grade reading. 31 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV this work. In this way, the Read activity is sustainable at the classroom, and, in turn, the school level. New teachers will follow the school culture. Because Read works at the school level, change can be more permanent. In this way UNIBE has been effective in making change for sustainability and engaging MINERD’s actors (i.e. teachers) in the process. At the same time, the role of MINERD technical staff is key. Technical staff at the district and regional levels have more recently been engaged in Read. This engagement must continue in order to build sustainability from the decentralized level up. Furthermore, it is important that technical staff and school pedagogical coordinators are technically strong on the approach to literacy. Again, more recently UNIBE has been successful in engaging the more decentralized level of MINERD (i.e. districts and regions). The only critique is possibly that it should have been done at the outset. Lastly, as discussed in question 6, the reading clubs are effective. They teach reading skills and promote a love of reading. The main ingredient at the local level to sustain the initiative of the reading clubs, is if they are convinced of the benefits of the clubs. Beneficiaries in field visits (parents, teachers, directors and students) are highly committed, interested in, and convinced of the benefits of the clubs and therefore likely to sustain them in the future. Figure: Sustainability Prospects As seen in the figure above, 66.6 percent of KII respondents either believed that Read will be sustained or were uncertain about its sustainability, while 33.3 percent of respondents did not believe or had no comment on whether Read approaches would be sustained. Respondents thought that there was still work to be done to ensure sustainability. There are prospects of sustainability at the local level, as indicated by the NGO DREAM, regarding sustained changes in teaching pedagogy and practice. With respect to engagement of senior Ministry officials to achieve sustainability of Read, the findings from the KIIs show that UNIBE has maintained ongoing conversations at the central level with senior MINERD officials, especially more technical directions - for example Curriculum and Primary. Yet at the USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 32 same time, more needs to be done, especially with other directions - for example, Planning (which inherently links finance and budgetary aspects to the planning and implementation of education programming nationwide, including for literacy). The findings of the interview conducted with the MINERD Vice Minister (discussed again under Question 9) show that the Ministry is focused on implementing their new, national teacher training strategy and perceives that Read is not aligned with this strategy. The new strategy works through improving education management, providing more support to teachers for effective use of the Ministry’s curriculum in reading and mathematics, and strengthening the District teams to provide mentoring. While it includes some changes to the curriculum, the strategy assumes that curriculum reform will not in itself improve learning outcomes; rather there needs to be stronger management within schools and in the Districts to provide more and better support to the teachers. The senior MINERD officials point out that USAID and UNIBE had engaged with MINERD but not INAFOCAM, and hence there was a missed opportunity for aligning approaches, methods and implementation arrangements. At this moment, Ministry leadership is not open to sustaining and expanding the Read approach. They took pains to clarify that when they term the Read approach as “complementary”, this should be interpreted as a permission to utilize it in the Read schools alongside of the Ministry’s approved curriculum, but it does not signify that it will be adopted formally by the Ministry. USAID and UNIBE clearly need to do more in this direction to reinforce MINERD support for Read going forward. In addition, according to reflections from MINERD, Read will continue through the use of the activity’s materials. MINERD has reviewed the materials (i.e., the leveled readers and decodables) and agreed that they will be used in MINERD’s nation-wide strategy as supplementary didactic materials. Senior officials indicated that they were more interested in the materials for later years (not years 1 to 3). In addition, a MINERD director noted the importance of the activity, MINERD, and teachers “walking together,” which addresses both levels of sustainability (on the ground and at higher levels). MINERD at all levels has largely been engaged for sustainability with regard to literacy, not parent participation (for example PTAs or clubs). These entities sustain via their own efforts at the school and local levels. See question 6. 33 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV H. QUESTION 8 To what extent has Read generated and shared knowledge of effective reading instruction? Stakeholders confirm that UNIBE is associated with high quality research. This was heard by NGOs and MINERD. In addition, UNIBE presents research internationally and in peer-reviewed journals. More recently, UNIBE has held knowledge-sharing events with EDUCA (November 2017) and with MINERD (November 2017). These events have served as a space to share findings from the first two years of the project. Lastly, UNIBE has held events with school directors to share results. These events have been more recent and should continue. At the same time, some observers have noted that, despite the high quality of the research, the materials presented are not user friendly. The images can be confusing and difficult to decipher. Given this, it is important for UNIBE to remember who its audience is. An important part of sustainability is convincing the prime partner (i.e., the government and, in this case, MINERD) of the benefits of the Read approach through evidence. Therefore, UNIBE’s role is to present and share the information in a way that is easily interpreted by civil servants and other stakeholders alike. One example of communications from the World Bank is shown below, and even this is somewhat difficult to interpret. QUESTION 8 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS UNIBE has produced high quality research UNIBE produced high quality research and has started to share with MINERD, although more is required (for example, the EDUCA congress/forum) UNIBE has produced school-level knowledge, although gaps remain in giving this to the school level for decision making (see World Bank) UNIBE has shared research with the international community UNIBE has shared less research with MINERD Recently there was an event where results were presented to MINERD UNIBE suggested that graphics have been shared with schools on performance (examples exist from the World Bank) Principals report participating in knowledge-sharing events, although they did not receive handouts (to date) USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 34 Photo: Example of Knowledge Sharing Materials for School Level I. QUESTION 9 How will MINERD’s teacher training strategy affect the Read activity? QUESTION 9 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS MINERD is moving ahead with its own teacher training and mentoring which includes changes in curriculum in early grade reading. It will be rolled out in all schools throughout the country. The new teacher training strategy should be considered a “co-intervention bias” caused by MINERD interventions in the same Read treatment and control groups of schools When MINERD does conduct training and mentoring in the Read schools, it could invalidate the impact evaluation design, because it will not be difficult to separate out the effects of the Read intervention. MINERD training operates by contracts with DR entities (INOFOCAM, POVEDA, PUCCM, etc.) regionally divided into departments. MINERD’s main concern is ensure that Read activities do not interfere with the new training strategy. It will coordinate with Read and the other DR entities. 35 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV The evaluation team conducted a KII with Denia Burgos, Vice Minister, Barbarita Herrera, General Director of Primary Education, as well as other stakeholders involved in teacher training. The supplemental interview with Denia Burgos and Barbarita Herrera was conducted in February 2018 whereas all other interviews for this performance evaluation were conducted between October and November 2017. The interview with the two senior MINERD officials focused mainly on Question 9, but as explained above, the findings from this interview are relevant for other evaluation Questions (Nos. 1, 2 and 7). The Vice Minister and General Director of Primary Education prefaced their response to Question 9 with an explanation of the origins of the Ministry’s new teacher training strategy implemented by INAFOCAM in partnership with UNICEF and using universities and non-profits as training providers. They explained that beginning in 2014 INAFOCAM began the design of the strategy, producing a technical report, “Marco para la Formación Contínua”. The strategy focused on reading and mathematics, but also education management, looking to strengthen support to teachers within the school setting and from MINERD’s District teams. In addition to training, it includes a component of teacher mentoring. The strategy was piloted in two Districts (one rural, the other urban, for a total of 120 schools). It was subject to an external evaluation, which showed improvements in student learning in reading and mathematics. The current Vice Minister was at that time Director of INAFOCAM and led the new teacher training strategy. She reports that she was not contacted by USAID or UNIBE in the design stage of Read. When she left her position in INAFOCAM and became Vice Minister at MINERD there was a transition process with MINERD officials but Read was not a topic in the transition briefings and meetings. She said that she did not believe that Read was a priority for MINERD during the previous government. After becoming Vice Minister, she was contacted by USAID, which was when she learned of the Read activity, which she characterized as a “pilot project”. While she thought it was “interesting”, she said that her focus since 2016 has continued to be the implementation of the Ministry’s new teacher training strategy. USAID and UNIBE have explained in detail the Read methodology to Ministry officials, but MINERD does not believe that it is “relevant to the Ministry’s present strategy”. When asked whether the Ministry’s position would be changed if the current impact evaluation of Read was to show a significant MINERD will not adjust the new teacher training strategy to avoid “co-intervention bias” in the Read treatment and control schools. While MINERD has termed Read methods as “complementary”, at this time it does not intend to support the Read approach and methods beyond the existing schools. MINERD would welcome being able to use the Read materials for later grades. Co-intervention creates a problem for Read’s existing impact evaluation design. It makes it difficult to measure the impact of Read interventions separate from the impact of the MINERD training. MINERD sees the Read methods as “complementary” to its existing curriculum, but does not intend to promote their expanded use, even if it shows improvement in reading outcomes. MINERD values the effort made by Read in creating materials for later grades (not grades 1 to 3). USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 36 improvement in early grade reading, the Vice minister emphasized that the Ministry had conducted evaluations of its own pilot project in the two Districts in 2013-2014 and had seen “positive impacts on reading”. As explained above in the response to Question 1, while the INAFOCAM teacher training strategy was in development, it had no impact on the Read activities during the first two years of the performance period. However, as the strategy is being rolled-out, it will inevitably conflict with Read. MINERD’s main concern now is “to avoid Read hindering the implementation of the teacher training strategy”. For this reason, the Ministry has given instructions to INAFOCAM to coordinate closely with UNIBE to ensure that it doesn’t interfere with the teacher training activities. The INAFOCAM training strategy assigns each training provider to a department or departments and assumes that all schools within a given department will be trained during the same period. There is no provision in the activity planning or the subcontracts with the training providers for exempting those schools in the Read treatment and control groups. While coordination of Read activities with INAFOCAP training providers is a relevant implementation issue, it is not the most pressing issue that emerges from the KII with senior Ministry officials. Rather, their responses go to the heart of the Read activity’s impact and sustainability. The Vice Minister and the General Director of Primary Education often referred to the Read approach and methods as a “complementary method” to be used by teachers in addition to the Ministry’s own curriculum and teaching methods. In response to the evaluation team’s follow-up questions about the practical implications of the perception of Read as a “complementary method”, the MINERD officials emphasized that the Ministry does not have the resources to train teachers in additional teaching methods, hence while teachers in the Read schools may continue to use the method to reinforce their teaching, the Ministry will not be conducting further training on the Read approach to early grade reading. The current strategy is costly and will take continuous effort over several years to implement. The one aspect of the Read activity that could be of interest for the Ministry going forward are the teaching materials, particularly those used by teachers in later grades (but not the materials for grades 1 to 3). “Last year the materials were reviewed by the Ministry, and UNIBE made major changes in response to our recommendations. They could be useful for us, subject to a technical assessment”. In responding to this evaluation question, the evaluation team also considered the possible effect of the INAFOCAM teacher training and mentoring on the Read activity’s quasi-experimental impact evaluation design. This design is a kind of randomized control trial (RCT), with schools (along with their respective teachers and students) assigned to treatment and control groups. This allows UNIBE to compare reading outcomes of students in schools receiving Read interventions (treatment) with outcomes in schools that have not received the interventions (control). Read’s theory of change is predicated on a robust evaluation design, which will create convincing evidence for the benefits of the Read approach and methods. In the literature on research bias, the training that INAFOCAM will be providing creates a type of “co￾intervention bias”, i.e., when some subjects are receiving other interventions (INAFOCAM training and mentoring) at the same time as the study treatment (Read training and mentoring). Such additional 37 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV interventions besides those that are formally included in the Read experiment may introduce confounding variables and erode the validity of the results of the trial. In this case, it will be difficult to separate out the independent impact of the Read interventions from the impacts of the INAFOCAM interventions. The problem is compounded by the fact that INAFOCAM training will be provided on a rolling basis over the remaining two years of Read implementation, with some of the Districts where Read treatment and control schools are located having longer “exposure time” to INAFOCAP training and mentoring interventions than others. This issue requires a careful methodological review before continuing with the evaluation component of the Read activity. J. QUESTION 10 What would the Ministry recommend in order to improve USAID programming? The entity responsible for designing and expanding MINERD’s strategy is INAFOCAM. The Institute contracts UNICEF to train the different entities that implement MINERD’s strategy in the districts (for example Sur Futuro, Pontificia Universidad Católica Madre y Maestra [PUCMM], possibly the Pedro Poveda Higher Institute of Educational Studies, etc.). In other words, INAFOCAM is responsible on the part of MINERD for teachers’ capacity formation, a crucial part of its strategy. In the words of INAFOCAM; “It would be great if UNIBE continues to pave the way and we build off of their approach.” In other words, the Ministry recommends that USAID continue to support literacy. This is especially the case in terms of providing concrete “how to” training approaches to teachers in schools. While MINERD has the strategy (the what) they recommend USAID support by providing examples and practical training (the how). This was confirmed with MINERD (assessment, curriculum and primary directions of the MINERD). MINERD noted that Read follows a series of USAID investments, including the Effective Schools Program (ESP) with PUCMM and, even prior to that, the Centers for Excellence in Teacher Training (CETT). While these programs promoted a whole language approach, MINERD notes that USAID continues to support reading in the Dominican Republic, and corresponding changes in reading pedagogy perspectives and orientations. The MINERD Vice Minister recommends that USAID review the Read QUESTION 10 FINDINGS AND CONCLUSIONS FINDINGS CONCLUSIONS The strategy consists of a change from competencies to content There is overlap and opportunity for Read support in literacy, management, and M&E in the strategy (not mathematics- possibly other donors) USAID needs to better align with MINERD policies. This alignment might include a change in Read scope of work for the second half of the performance period. The content focuses on reading/writing, mathematics, leadership/ management and monitoring and evaluation INAFOCAM has designed a long-term strategy for teacher training. The Ministry recommends that USAID review current Ministry policy and align its education programs accordingly. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 38 activity and seek to align it with Ministry priorities going forward. The Ministry is open to consultation on this point. Another clear demand from MINERD is for more instruction and focus on writing. This was noted in the textbox in section C. Given the title of the project is “Read,” it is interesting that MINERD wants to explore how the activity and the ministry can focus on writing skills in addition to reading skills for the remaining two years. Leading linguists and academics (for example Catherine Snow, Patton Tabors, Harvard) argue that reading and writing are interrelated and go hand in hand when it comes to teaching￾learning to read/write. V. RECOMMENDATIONS This section presents seven recommendations. In each recommendation, specific details are provided that link the recommendation to the conclusions articulated in the previous section. RECOMMENDATION 1: ALIGN READ WITH MINERD PRIORITIES. Despite Read’s progress during the first two years, at present MINERD is not committed to sustaining or extending the Read approach. The Ministry is now focused on implementing its own strategy for teacher training, which incorporates some curriculum changes in early grade reading. The Mission should explore with MINERD how to better align Read with the Ministry’s strategy for teacher training in early grade reading. This could require a change in the scope of the cooperative agreement with UNIBE, to focus on activities of interest to the Ministry. One of these areas may be mentoring, in which Read has shown good results. RECOMMENDATION 2: ASSESS READ’S IMPACT EVALUATION DESIGN The evaluation team finds that the INAFOCAM teacher training and mentoring activity will likely interfere with the Read activity’s impact evaluation design. The INAFOCAM activity constitutes a major “co-intervention bias” which will impede UNIBE from measuring the independent impacts of the Read interventions. The evaluation team recommends that USAID assess the Read impact evaluation design to confirm the possibility of a co-intervention bias, and if confirmed, suspend this component of the Read activity. In this case, USAID should consider other evaluation activities to support INAFOCAM’s current teacher training strategy. RECOMMENDATION 3: SHARE FINDINGS Whatever the decisions about the Read activity’s continuity, it should share findings in national forums, especially with MINERD, at the national, district and school levels. This reflects findings and conclusions related to Question #8. It is important that the Read activity share findings in a user-friendly way so that stakeholders fully understand and have ownership of the findings in their schools, which can ultimately support efforts for sustainability in the long term. RECOMMENDATION 4: CONTINUE READING CLUBS 39 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV Again, depending on how the Read activity is realigned with MINERD priorities, it should continue and, if the opportunity exists, expand the reading clubs. Aligned to the Read approach, in considering Question #6, the reading clubs were found to increase involvement, improve skills, and encourage a culture of reading. The clubs are well received in schools and they serve as an optimal space in which parents can get involved in a practical way to help their children learn and love to read. To this end, we recommend increased community resources, for example from the APMAES. RECOMMENDATION 5: CONTINUE THE EMPHASIS ON MENTORING In realigning the Read activity, USAID should ensure that MINERD can benefit from Read’s success in the mentoring component. Per the systematic qualitative evidence collected related to Question #4, mentoring strengthens school coordinators and district technicians to perform their role and is considered the most effective aspect of the Read program. To implement this recommendation in the second half of the activity with sustainability in mind, the Read activity should begin now to determine who will assume mentorship roles in the future, in particular school coordinators and district technical staff with a specialization in Spanish language and reading and bring these individuals on board. RECOMMENDATION 6: RETHINK SAFE SCHOOLS The Read activity’s approach to working with schools in the context of crime and violence, especially against women and girls, needs to be rethought in any attempt at realignment with MINERD priorities. Per responses to Question #5, the Safe Schools intervention’s strategy and implementation approach needs to be assessed. The intervention could, for example, have a bigger impact working at the awareness-raising campaign or policy levels, or could include a pilot on gender and social inclusion. The Read activity could look to existing, successful cases considering the context of the Dominican Republic, for example UNICEF’s work in Boca Chica, as a model for Safe Schools. RECOMMENDATION 7: GET IT IN WRITING. Based on findings related to multiple questions, the Read activity should get approvals and agreements in writing from MINERD, especially regarding new materials. This is important given the high potential for turnover and the varied power structures within MINERD. Any realignment of Read with MINERD’s new strategy should also be based on written agreements. The Read activity should share findings in national forums, especially with MINERD, at the national, district and school levels. This reflects findings and conclusions related to Question #8. Importantly, it is important that the Read activity share findings in a user-friendly way so that stakeholders fully understand and have ownership of the findings in their schools, which can ultimately support efforts for sustainability in the long term. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 40 ANNEX A. GENDER AND SOCIAL INCLUSION This annex examines gender and social inclusion as a cross-cutting theme throughout the evaluation. The questions below were added to the evaluation questions where gender and social inclusion were relevant issues. The evaluation team looked at gender and social inclusion with regard to materials (question 2); gender and social inclusion with regard to teacher formation (question 3); gender within the context of the Safe Schools intervention (question 5); and lastly, gender and social inclusion in the case of parents and community participation (question 6). • Q2 (reading strategies, resources and materials) – Is a gender lens applied to the materials created by UNIBE (and/or other socially￾excluded groups)? How? UNIBE leadership appears to be very conscious of gender and social inclusion at the central level. This is apparent not only in the standard approach to integrating gender and inclusion through disaggregating indicators, but also in programmatic aspects such as material development. UNIBE has promoted the revision of its own materials as well as those of MINERD to ensure that stereotypes are not presented. For example, in UNIBE’s own levelled readers, it took care to ensure that there is diversity and a variety of gender roles. One example of this was in the saga or series UNIBE developed. The topic was to be “your own super hero” and this was applicable to girl and boy readers. To create these materials, UNIBE worked with a team of illustrators. KIIs reflected that UNIBE learned from this experience as well and self-reflected on gender overall for the activity. • Q3 (teacher training) – Do teachers have different perspectives on girls’ and boys’ (and/or other socially￾excluded groups) performance and achievement? As seen in the figure on the following page, teachers do not appear to give differential treatment to girls and boys in the classroom. Sixty-five percent of classroom observations indicate that girls and boys are treated equally. If there is a difference, then the difference appears to favor girls. In 30.4 percent of classrooms, teachers called on girls more frequently; only in 4.3 percent of observations were boys called on more frequently (i.e., in one classroom). 41 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV Figure: Opportunities in the Classroom Girls vs. Boys In addition, the evaluation team saw that teachers make some modifications for students’ learning needs. Sixty-one percent made some modification to their teaching to help at least one student’s learning needs. This may include one-on-one support, using visuals, or another method. On the other hand, 39.1 percent made no modifications for learning needs. In the same respect, 60.9 percent of teachers addressed discipline (for example came closer to a student’s desk to keep the student on task or verbally directed the student’s attention), whereas 39.1 percent did not. It is possible that there were no discipline needs during the classroom observations, but qualitative data suggests that some teachers do not know how to handle discipline problems and/or learning needs and therefore do not address them. Figure: Discipline Figure: Learning Needs USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 42 The Read activity’s management team was trained using the GIS guide and they trained the teachers (as well as psychologists). It is unclear the extent to which the GIS guide was integrated into trainings across the board related to reading. According to mentors, more can be done to coordinate between Reading and GIS Mentors, since they are all part of the same project and their work in schools is directly related. For example, violence or conflict in the home can lead to social/emotional challenges which have repercussions for learning to read in the classroom (i.e., concentration, etc.) • Q5 (safe schools) – How effective is gender and social inclusion promoted in safe schools? To supplement the description of Safe Schools and related findings and conclusions in the body of this evaluation report, the team notes here a few important aspects related to gender and social inclusion in the Safe Schools intervention. First, the Read activity was advised to stop talking about gender equality by school level actors (i.e. directors). The topic was taboo and often was interpreted as related to sexual diversity (i.e., including gay, lesbian, bisexual, transgendered persons). This is representative of even more conservativism on the part of schools and communities in the country. • Q6 (parent/community participation) – How does participation vary by gender- mothers/ fathers, grandmothers/ grandfathers (and other socially-excluded groups), etc.? As described in the discussion on Question #6, parents participate in reading clubs and in APMAEs as part of the Read activity. In the 10 guided conversations the evaluation team held with parents, they noted their participation in reading in school and in the home (for example, helping with homework and reading to their children). Additionally, that it tends to be mothers who participate in their children’s education. In stark contrast to the larger proportion of mothers who participate in their children’s education, the team noted that the majority of the heads (i.e. the Presidents) of the APMAEs were males. Therefore, while mothers more frequently come to meetings and support reading in the home, fathers are provided the opportunity to serve as the leader. This is important to note in terms of the types of stereotypes with regard to leadership that are being perpetrated by these school structures. 43 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV ANNEX B. STATEMENT OF DIFFERENCES USAID Evaluation Policy allows for a statement of differences to be added to evaluation reports if a team member or a major stakeholder does not agree with one or more findings, conclusions, or recommendations. In this case, the statement of differences comes from one of the evaluation team members, Dr. Marcelo Castrillo, and is based on the results of interviews with teachers and school Directors in Read treatment group schools. The differences refer to two aspects of Read: the teachers’ understanding and utilization of the Read methods; and the potential impact of the Read mentoring on performance. Understanding and Utilization of Read Methods One of the findings of this evaluation of Read implementation has been the resistance of MINERD to the use of Read’s teaching materials. As noted above in the body of the report, in response to Questions 2 and 3, MINERD has resisted the implementation of Read methods, and refers to Read as a “complementary” approach. Other stakeholders interviewed have referred to the Read intervention as being “sandwiched” into the existing curriculum in the schools. Indeed, through the first two years of the project this evaluation found that the Ministry has not allowed the teachers to use the materials developed by Read. In the ten schools visited by Dr. Castrillo, none of the teachers or Principals could explain the main differences between the Read approach and the Ministry’s curriculum for early grade reading. The most common response to the question was that “there is no major difference”. In this sense, Dr. Castrillo’s own field work leads him to qualify the findings and conclusion in the response to Question 3, Has the training offered by Read helped teachers integrate the reading strategies effectively? The interviews in the ten schools suggest that the integration of Read methods into classroom teaching in the Read treatment group of schools still has some ways to go. Impact of Read Mentoring on Teacher Performance This incipient up-take of Read methods in the schools could well result in a lower than expected impact of Read interventions (this will need to be confirmed by the impending impact evaluation to be conducted by UNIBE). When asked, then, what is the main contribution of Read to improvement of reading? all of the teachers and Principals interviewed by Dr. Castrillo referred to the mentoring (“acompañamiento”) and the feedback as the most valuable element of the Read activity. Indeed, one of the most noteworthy findings about the evaluation of the Read intervention is the enthusiastic comments from all teachers and Principals for the Read mentoring component. It is not clear from the interviews, however, whether the mentoring is valued for its role in reinforcing the Read methods, or for giving teachers support and positive reinforcement in an institutional context which has been lacking in this aspect. From the interviews, it was clear that the teachers felt “proud” of being part of the Read project. They mentioned, too, a “competitive” dimension of the Read activity, which would generate information USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 44 about the performance of their schools versus others. This perception appeared to be an important motivating force for the ten teachers interviewed by Dr. Castrillo. The question is what role it might have in improving teaching and reading outcomes. Would it account for some of the improvement in reading? From these interviews with teachers and Principals, Dr. Castrillo puts forward the possibility that some of the Read activity’s potential impacts on students’ reading and comprehension skills could be caused by an observation effect, commonly called the “Hawthorne effect” in which “[a]wareness of being observed or having behavior assessed engenders beliefs about researcher expectations. Conformity and social desirability considerations then lead behavior to change in line with these expectations”.4 In this case, the effect would be due to the coordinators’ and mentors’ constant attention to observing and mentoring the teachers; and, the knowledge that their students care being assessed constantly and compared to students in other schools. This explanation is plausible, given the still incipient understanding of Read methods as found in the interviews with the teachers and Principals in the ten schools. Conclusions and Recommendations With respect to the integration of Read methods into the curriculum, Dr. Castrillo’s findings suggest that the Read methods have limited up-take by teachers, who do not fully understand the approach, and may not be using the methods fully. Thus, with regard to the Read activity’s mentoring component, it is possible that the improved mentoring has its own positive effects, irrespective of the teaching method being used. The recommendation that follows is that if feasible within the existing organization and budget of the Read activity, UNIBE should consider creating a sub-group of schools receiving only the monitoring and mentoring component, without changing the teaching methods. This would allow UNIBE to measure the independent effects of the teacher mentoring using the Read approach and technologies. This recommendation is especially relevant given the MINERD’s continuing resistance to the Read approach. 4 J. McCambridge, et al. 2010. “Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects”, Journal of Clinical Epidemiology. 2014 Mar; 67(3): 268. 45 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV ANNEX C. RESOURCE LIST • LEER/USAID (2015) Cooperative agreement and program modifications • LEER/USAID (2015, 2016, 2017) Annual Work Plans • LEER/USAID (2015) MEL Plan • LEER/USAID (2015, 2016, 2017) Annual Reports • LEER/USAID (various) Quarterly Reports • LEER/USAID (2015) Baseline Report • LEER/USAID (2015) Year 1 end of second grade report • LEER/USAID (2016) Year 2 end of second grade report • LEER/ USAID (2015) Performance Indicators • LEER/USAID (2016) Baseline Report for the Gender, Inclusion and Safe School Components of Read • LEER/USAID (2016) Community Baseline Report USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 46 ANNEX D. KII LIST Date Institution Person (s) 11-01-2017 USAID Francesca Hernandez, AOR 10-31-2017 MINERD Ancell Sheker, M&E/ Ass. 11-1-2017 UNIBE Sonia Molina, training 11-1-2017 UNIBE Cledenin Veras, literacy 11-2-2017 Dream Catherine Delaura, Director Rebecca Martinez, Literacy 11-2-2017 Fe y Alegria Jesus Zagulul, Director 11-2-2017 MINERD, Dir. Primary Dir, Primer Ciclo Barbarita Herrera Genny Luz Gómez 11-2-2017 EDUCA Patricia Mejía Alejandro Rodríguez Katherine Javier 11-3-2017 UNIBE, CoP Laura Sartori 11-3-2017 UNIBE, M&E Aida Mencia 11-3-2017 UNIBE Coordinators (2) Mentors (5) 11-8-2017 INAFOCAM, Coordinator Dominga Comas 11-8-2017 World Bank, Governance Specialist Carmen Amaro 11-8-2017 World Vision Maratini Traveras 11-8-2017 USAID, Office Chief USIAD, COR Alerta Joven Kimberly Ludwig Mariela Pena 11-9-2017 Fun. Inicia, Director Antonio Caparos 11-10-2017 PUCMM, Researcher Liliana Montenegro 11-10-2017 MINERD, Planning Remijio Garcia 11-13-2017 UNICEF Lisette Nunez 11-14-2017 Sur Futuro Bernarda Firmo 47 | FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY USAID.GOV ANNEX E. FIELDWORK SCHOOLS USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 48 ANNEX F. LIMITATIONS Constraint/Bias Result Mitigation Strategy The evaluation team will choose only 24 schools in the Duarte Corridor in which to conduct data collection. Selection bias. Capacity may vary across schools influencing Read effectiveness. Team members will look at monitoring data that is disaggregated by site to predict differences between schools and remoteness. Within the schools that will be visited, schools will be considered for varying degrees of accessibility. Read beneficiaries may respond to questions posed by the evaluation team with answers that blend their experiences into a composite memory. Beneficiaries who may have participated in more than one activity or in similar activities conducted by several implementers may not distinguish them as separate activities. Recall Bias. Experiences may be misattributed to Read activities. The evaluation team will employ strong data collection tools that specify exactly the intervention being evaluated. The evaluation team will also triangulate data, matching information from different sources to corroborate varied experiences. A respondent may provide positive or negative remarks about an activity because s/he would like to attend such trainings in the future; or implementing partners may believe that evaluation findings could result in reduced assistance from USAID/DR. Response Bias. More likely some responses will be skewed to more positively stating Read successes The team will be especially cognizant of the risk of response bias in cases when MINERD representatives accompany team members during FGDs and KIIs. The data collection protocol will clearly state that answers will not have any effect on an individual’s or school’s receipt of future services. The team will triangulate data from field work with monitoring data or third￾party data. The evaluation team may be led to beneficiaries with more positive experiences, for example Read partners may select the more active, responsive, or engaged respondents. Selection Bias. There is a risk that the team may only hear from key informants who report positive experiences. To mitigate this risk, the evaluation team will work closely with UNIBE in respondent selection, using participant lists wherever possible. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 49 ANNEX G. DATA COLLECTION INSTRUMENTS This Annex contains the data collection instruments, first in English, then in Spanish. Data collection instruments are: • KII protocol for stakeholders • KII protocol for school staff (principals and teachers) • Parent FGD protocol • Classroom observation tool USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 50 KEY INFORMANT INTERVIEW - STAKEHOLDERS Name: Date: Role: Organization: Introduce self, purpose of study, and confidentiality. 1. Please reflect on Read implementation. Have there been the major implementation challenges in the cooperative agreement?5 Probe… (please say more) 2. Please reflect on reading strategies. How has UNIBE integrated different strategies and resources with teacher training and therefore into classroom instruction?6 Probe… (please say more) 3. Please reflect on training. Has the training offered by Read helped teachers use effective strategies within MINERD’s new curriculum?7 Probe… (can you share more) 4. Please reflect on mentoring. Has teacher mentoring been effective? Probe… (please share how) 5. Please reflect on school safety. How has school safety changed since the intervention? Probe… (please share how) 6. Please reflect on community involvement. Are the Clubes Leyendo supported by UNIBE and World Vision effective? Probe… (is there more you can say) 5 In this question probe what is the project like in your school/ classroom? Please share more… 6 In this question probe how do you teach reading in your school/ classroom? 7 In this question probe: what was the training like, how was it helpful? Please give examples… USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 51 7. Please reflect on sustainability with MINERD. How successful is Read in incorporating MINERD in the design for long term sustainability?8 8. Please reflect on knowledge management. To what extent has Read generated knowledge about effective reading instruction and shared it with stakeholders? 9 Thank you. If there is more you would like to share, please feel free to do so. 8 In this question probe about how the project can be sustained by the ministry, will the project continue, how? 9 In this question ask: what kind of reports have been shared? What did you learn? Useful?... USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 52 KEY INFORMANT INTERVIEWS - SCHOOL PRINCIPALS AND TEACHERS Name: Date: Role: School, location: Introduce self, purpose of study, and confidentiality. 1. Please reflect on Read implementation in your classroom and/or school. Have there been the major implementation challenges? Probe… (please say more) 2. Please reflect on reading strategies. How has UNIBE integrated different strategies and resources with teacher training and therefore into classroom instruction? Probe… (please say more) 3. Please reflect on training. Has the training offered by Read helped teachers use effective strategies with the MINERD curriculum? Probe… (can you share more) 4. Please reflect on mentoring. Has teacher mentoring been effective? Probe… (please share how) 5. Please reflect on school safety. How has school safety changed since the intervention? Probe… (please share how) 6. Please reflect on community involvement. Are the Clubes Leyendo supported by UNIBE and World Vision effective? Probe… (is there more you can say) 7. Please reflect on sustainability with MINERD. How successful is the activity in incorporating MINERD in the design for long term sustainability? 8. Please reflect on knowledge management. To what extent has Read generated knowledge about effective reading instruction and shared it with stakeholders? Thank you. If there is more you would like to share, please feel free to do so. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 53 PARENT FOCUS GROUP DISCUSSION GUIDE School: Location: Date: Number of participants (males, females): Socio-economic status of participants: Introduce self, purpose of study, group respect, and confidentiality Question 1. How are parents and the community involved in school, education and literacy? Question 2. Are the Clubes Leyendo supported by Read effective? Question 3. What is the approach? Is it enough? Is the time of day appropriate? Question 4. Are parents and the community involved? How are they involved? Has involvement increased or decreased? How? Question 5. Has Read support had an effect on the APMAES (parent associations)? Question 6. Have parents and students participating in Clubes Leyendo seen any changes in their children’s literacy? How? Question 7. Is there more you would like to share? Thank you and closure. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 54 CLASSROOM OBSERVATION TOOL BASIC INFORMATION School: Code: Morning/ Afternoon: Location: Date: Time: Observer: TEACHER DATA Grade: Teacher name: Number of students: Gender: Years of experience: How long in the school: How long in Read project: Level of education: CONTEXT MATERIALES/ AMBIENTACION 1. Teacher guide: _____ 1= the teacher has a teacher guide and has consulted it 2= the teacher has a teacher guide and does not consult it 3= the teacher does not have a teacher guide Observations: 2. Teacher plan: ____ 1= the teacher has a lesson plan and refers to it heavily 2= the teacher has a lesson plan and refers to it on occasion 3= the teacher has a lesson plan, but does not use it 4= the teacher does not have a lesson plan Observations: 3. Student texts (mathematics only): _____ 1= all the students have textbooks (100 percent) 2= more than half the students have textbooks (50-99 percent) 3= less than half the students have textbooks (0-49 percent) USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 55 4= there are no textbooks (0 percent) 3a. Students use the texts in the lesson (yes/no): ____ Observations: 4. There is a biblioteca de aula (yes/ no): _____ Note: the biblioteca de aula is a donation under the PEF program of 70 books at the grade level of the class. BEP has donated materials to MOST schools. 4a. About how many books are in the biblioteca? _____ 4b. Do students use the biblioteca de aula in the lesson (yes/no)? ______ Observations: 5. There are other materials in the classroom (yes/no): ____ Note: This could be education materials from a store, recycled materials from home, additional books other than the biblioteca. 6a. What are they? 6b. Are they being used in the lesson (yes/no): ______ 6. Classroom education materials created by students and teachers on the wall: ____ 1= the walls are covered with education related materials that are age appropriate (this implies that they are inviting to the students, i.e., colorful) 2= the walls have some education materials, but they may not be age appropriate or not accessible to students in the class, or they may not be education related 3= there is little to no education related material on the walls 7a. How many of the walls available are covered? ___ Observations: 7. There are rincones de aprendizaje in the classroom (yes/no) _____ USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 56 Note: a rincón de aprendizaje is a small space that is established where various resources are established for students to work independently or in small groups and learn about a particular theme. There may be books, materials, posters, etc. 8a. How many? ____ 8b. What area? ____ 8c. Are students using it? ____ Observations: 8. How are the students seated? INSTRUCTIONS: Draw classroom on back; identify teacher 9. How are the students working in the class? ___ a= the teacher stands at the front of the class and the students are in their seats b= students are in pairs working c= students are in small groups working d=students are not working 10. What is the percentage of students engaged in learning activities? ___ 1= all the students are engaged (100 percent) 2= over half the students are engaged (50-99 percent) 3= less than half the students are engaged (0-49 percent) 4= none of the students are engaged (0 percent) Observations: CLASSROOM PROCESSES 11. Active teaching methodology/ constructivist approach Note: an active teaching-learning methodology as defined by the Read program entails reflection on students’ previous knowledge, constructing knowledge, actively using materials, reviewing what is learned and applying the knowledge. 12a. The teacher engages students in a dialogue (yes/no): _____ 12b. Students demonstrate knowledge with materials (yes/no):_____ 12c. Students have opportunities to practice what they learn (yes/no):_____ 12d. The students apply what they are learning (yes/no): ____ USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 57 Observations: 12. The teacher uses the following techniques (observations to right): The teacher 13a. The teacher writes on the board To provoke a discussion____ To have students answer questions_____ To have students copy_____ To summarize____ To have students read____ 13b. The teacher verbally Dictates____ Reads aloud ____ Asks students questions (individually)____ Asks students questions (as a group) ____ 13c. The teacher physically demonstrates_____ 13d. The teacher physically demonstrates with materials____ 13e. The teacher draws___ 13f. The teacher uses a poster____ The students 13g. The students read Aloud____ Silently____ 13h. The students write on the board ____ 13i. The students role play____ 13j. The students demonstrate physically_____ 13k. The students demonstrate with materials_____ 13l. The students do seatwork_____ 13m. Other _____ USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 58 13. Reinforcement techniques (choose all that may apply): 14a.The teacher explicitly thanks students for participation____ 14b. The teacher reinforces with a positive expression_____ 14c. The teacher is neutral____ 14d. The teacher criticizes individual students_____ 14e. The teacher criticizes the whole class____ 14. The teacher moves to address students’ discipline needs (yes/ no)____ 15. The teacher moves to address students’ learning needs (yes/no) _____ 16. Gender analysis: girls and boys participation______ INSTRUCTIONS: Observe for five minutes and count the number of times the teacher calls on girls/ boys in the table below 1= girls and boys are given fairly even opportunities to participate 2= girls are given more opportunities to participate 3= boys are given more opportunities to participate 4= there is little to no participation Girls Boys Observations: USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 59 ENTREVISTA INFORMANTES CLAVE - GRUPOS DE INTERÉS Nombre: Fecha: Papel o cargo: Organización: Preséntese a usted mismo/a. Indique el propósito del estudio y las pautas de confidencialidad. 9. Por favor, analice la implementación de LEER. ¿Estaban contemplados en el acuerdo de cooperación los principales desafíos de implementación? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 10. Con relación a las estrategias de lectura, ¿cómo considera que UNIBE ha integrado las diferentes estrategias y recursos con la formación de docentes y, por lo tanto, con la enseñanza en el aula? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 11. Hablemos sobre la capacitación. ¿Hasta qué punto, cree usted, que la capacitación ofrecida por LEER puede haber ayudado a maestros y maestras a utilizar estrategias más efectivas dentro del nuevo plan de estudios del MINERD? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 12. Con respecto al acompañamiento, ¿cómo considera usted que ha sido la efectividad del mismo? ¿Hasta qué punto el acompañamiento ha sido una estrategia efectiva? Coménteme, a su juicio, cuáles han sido los pros y los contras del acompañamiento. Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 13. Ahora, tratemos el tema de la seguridad escolar. ¿En qué manera cree usted que la intervención de LEER ha influido sobre la seguridad escolar? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 14. Y conversemos sobre la participación de la comunidad. ¿Cuál considera usted que ha sido la efectividad de los Clubes Leyendo, implementados con el apoyo de UNIBE y Visión Mundial? ¿Cuáles pueden haber sido sus puntos fuertes y sus puntos débiles? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 15. Un tema muy importante, la sostenibilidad del proyecto con el MINERD. ¿Hasta qué punto usted considera que LEER ha tenido éxito, en el incorporar el MINERD en el diseño del proyecto, para garantizar la sostenibilidad a largo plazo? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 60 16. Reflexionemos sobre la gestión del conocimiento. ¿En qué medida LEER generó conocimiento acerca de la enseñanza efectiva de la lectura y la compartió con las partes interesadas? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 17. Y por último, ¿cómo considera que LEER debería coordinarse con otros implementadores que trabajan con la estrategia de capacitación docente del MINERD? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” Gracias. Si hay más que quisiera compartir, siéntase libre de hacerlo. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 61 ENTREVISTA INFORMANTE CLAVE - DIRECTORES Y DOCENTES DE LA ESCUELA Nombre: Fecha: Papel: Escuela, Ubicación: Preséntese a usted mismo/a. Indique el propósito del estudio y las pautas de confidencialidad. 1. Hablemos sobre la implementación de LEER en su salón de clase y / o en esta escuela. ¿Cuáles han sido los principales desafíos o retos para su implementación? ¿Cuáles cree que han sido las principales dificultades en su escuela? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 2. Con relación a las estrategias de lectura, ¿cómo considera que UNIBE ha integrado las diferentes estrategias y recursos con la formación de docentes y, por lo tanto, con la enseñanza en el aula? ¿Cómo se integraron esas estrategias en su escuela/aula? Seguimiento: En su implementación en el aula, ¿percibe usted diferencias en cuanto al impacto de la estrategia en el aprendizaje en niñas, con relación a los niños? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 3. Hablemos sobre la capacitación. ¿Hasta qué punto, cree usted, que la capacitación ofrecida por LEER puede haber ayudado a maestros y maestras a utilizar estrategias más efectivas dentro del nuevo plan de estudios del MINERD? (a director/a) ¿Cómo ha sido la capacitación en su escuela? (A maestro/a) ¿Cómo fue la capacitación que usted recibió? Seguimiento: En su experiencia, ¿hay diferencias con respecto al género, en la capacidad de asimilar y aplicar la capacitación, entre las maestras y los maestros? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 4. Con respecto al acompañamiento, ¿cómo considera usted que ha sido la efectividad del mismo? ¿Hasta qué punto el acompañamiento ha sido una estrategia efectiva? Coménteme, a su juicio, cuáles han sido los pros y los contras del acompañamiento. ¿Cómo ha sido el acompañamiento en su escuela/aula? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 5. Ahora, tratemos el tema de la seguridad escolar. ¿En qué manera cree usted que la intervención de LEER ha influido sobre la seguridad escolar? ¿Cómo se ha implementado esto en su escuela/aula? Seguimiento: Los problemas de seguridad, ¿impactan en manera diferente en estudiantes, según sea su género? ¿Qué diferencias percibe usted en la forma como el Programa LEER influye en la seguridad escolar, según el sexo del o la estudiante? USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 62 Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 6. Y conversemos sobre la participación de la comunidad. ¿Cuál considera usted que ha sido la efectividad de los Clubes Leyendo, implementados con el apoyo de UNIBE y Visión Mundial? ¿Cuáles pueden haber sido sus puntos fuertes y sus puntos débiles? Seguimiento: ¿Cuáles diferencias usted observa en la receptividad y apoyo a los Clubes Leyendo, al comparar las madres con los padres? Profundice con expresiones como “por favor, diga más sobre…” “explíqueme que significa para usted…” 7. Un tema muy importante, la sostenibilidad del proyecto con el MINERD. ¿Hasta qué punto usted considera que LEER ha tenido éxito, en el incorporar el MINERD en el diseño del proyecto, para garantizar la sostenibilidad a largo plazo? 8. Reflexionemos sobre la gestión del conocimiento. ¿En qué medida LEER generó conocimiento acerca de la enseñanza efectiva de la lectura y la compartió con las partes interesadas? Gracias. Si hay más que quisiera compartir, siéntase libre de hacerlo. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 63 GUÍA DEBATE DEL GRUPO DE ENFOQUE PADRES Escuela: Ubicación: Fecha: Número de Participantes (hombres, mujeres): Estatus Socioeconómico de los participantes: Preséntese a usted mismo/a. Indique el propósito del estudio y las pautas de confidencialidad. Pregunta 1. ¿Como consideran ustedes que están padres y madres, y la comunidad, involucrados en la escuela, educación y alfabetismo? Pregunta 2. ¿Conocen ustedes los Clubes Leyendo? Recoja las reacciones y diga: ¿De qué forma consideran que el apoyo que ha dado el programa LEER a los Clubes Leyendo, ha sido efectivo? Pregunta 3. ¿Cómo consideran la forma en que este programa se desarrolla? ¿Hasta qué punto esto es suficiente? ¿Cómo consideran la hora del día en que se ofrece? Para ustedes, ¿esta es una organización apropiada? ¿Porqué? Pregunta 4. ¿Están involucrados los padres y la comunidad? ¿Cómo están involucrados? ¿Ha aumentado o disminuido la participación? ¿Cómo? Pregunta 5. ¿Cuál efecto ha tenido apoyo de LEER en la APMAE (Asociación de Padres, Madres, Tutores y Amigos de la Escuela)? Pregunta 6. ¿Han visto los padres y estudiantes que participan en Clubes Leyendo algún cambio en la alfabetización de sus hijos? ¿Cómo? Pregunta 7. ¿Hay algo más que quisieran ustedes compartir? Gracias y cierre. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 64 HERRAMIENTA DE OBSERVACIÓN EN EL AULA INFORMACIÓN BÁSICA Escuela: Codigo: Mañana / Tarde: Ubicación: Fecha: Hora: Observador/a: DATOS DEL PROFESOR Grado: Nombre del Docente: Número de Estudiantes: Sexo: Años de Experiencia: Tiempo en la Escuela: Tiempo en el Proyecto LEER: Nivel de Educación: Materiales de contexto / Ambientación Guía del/la docente: _______________ 1 = el/la docente tiene una guía de docente y ha consultado 2 = el/la docente tiene una guía de docente y no consultar 3 = el docente no tiene una guía de docente Observaciones: Plan maestro: _______________ 1 = el/la docente tiene un plan de lección y se refiere a él fuertemente 2 = el/la docente tiene un plan de lección y se refiere a en ocasiones 3 = el docente tiene un plan de lección, pero no la utiliza 4 = el/la docente no tiene un plan de lección Observaciones: Textos de estudiante: _______________ 1 = todos los alumnos tienen libros de texto (100%) 2 = más de la mitad los alumnos tienen libros de texto (50-99%) 3 = menos de la mitad de los alumnos tiene libros de texto (0-49%) 4 = hay no hay libros de texto (0%) USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 65 Observaciones: Estudiantes utilizan los textos de la lección (sí/no): _______________ Observaciones: ¿Hay una biblioteca de aula? (sí / no): _______________ ¿Cuántos libros hay en la biblioteca? (aproximadamente)________________ ¿Estudiantes utilizan el biblioteca de aula durante la clase? (sí/no)?______________ Observaciones: ¿Hay otros materiales en el aula? (sí/no): _______________ Nota: Esto puede ser materiales educativos adquiridos, materiales reciclados del hogar, más libros que no sean de la biblioteca. ¿Qué son estos materiales? _______________________________________________ ¿Se están utilizando los materiales adicionales en la clase? (sí/no): _______________ Materiales de Educación de aula creados por estudiantes y docentes en la pared: _______________ 1 = las paredes están cubiertas con materiales relacionados con educación, que son de edad apropiada (esto implica que invitan a los estudiantes, es decir, llamativos, atractivos…) 2 = las paredes tienen algunos materiales educativos, pero no son apropiados para la edad, o no accesibles a los estudiantes en la clase, o no están relacionados con la educación 3 = hay pocos a ningún material educativo en las paredes Observaciones: ¿Cuántas de las paredes disponibles están cubiertas con materiales? ________________ Observaciones: ¿Existen rincones de aprendizaje en el aula? (sí/no) Nota: un rincón de aprendizaje es un espacio pequeño que se establece donde se dispone de diversos recursos para los estudiantes a trabajar independientemente o en grupos pequeños y donde aprenden sobre un tema en particular. Puede haber materiales, carteles, libros, etcétera. ¿Cuántos Rincones hay? ______________________ ¿Qué áreas o temas presentan? _____________________ ¿Son usados por estudiantes durante la clase? (sí/no)_____________ Observaciones: ¿Cómo se disponen los estudiantes en el aula, cómo están sentados? INSTRUCCIONES: Dibuje el aula en el dorso; identifique la posición que ocupa el/la docente ¿Cómo están trabajando los alumnos en la clase? a = el/la docente se encuentra en la parte delantera de la clase y los estudiantes está en sus asientos b = los estudiantes están en pares de trabajo c = son los estudiantes en pequeños grupos de trabajo USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 66 d = los estudiantes no están trabajando en la clase ¿Cuál es el porcentaje de estudiantes que participan en las actividades de aprendizaje? 1 = todos los estudiantes están participando (100%) 2 = La mitad o más de los estudiantes está participando (50-99%) 3 = menos de la mitad de los estudiantes está participando (1-49%) 4 = ninguno de los estudiantes está participando (0%) Observaciones: PROCESOS DE AULA Metodología Activa / enfoque constructivista Nota: una metodología activa de enseñanza-aprendizaje como se define por el Programa LEER implica una reflexión sobre el conocimiento previo de los estudiantes, construir conocimiento, uso activo de materiales, revisar lo que se aprende y aplicar los conocimientos. docente involucra a estudiantes en un diálogo (sí/no): _______________ estudiantes demuestran conocimiento con materiales (sí/no): _______________ estudiantes tienen oportunidades para practicar lo que aprenden (sí/no): _______________ estudiantes aplican lo que aprenden (sí/no): _______________ Observaciones: El/la docente utiliza las siguientes técnicas (observaciones a la derecha): El/la docente el/la docente escribe en la pizarra: para provocar una discusión _____ para hacer que estudiantes respondan a preguntas _____ para hacer que los estudiantes copien _____ Para resumir _____ Para hacer leer a los estudiantes _____ el docente verbalmente Dicta_____ Lee en voz alta_____ Pide a los estudiantes preguntas (individual)_____ Pide a los estudiantes preguntas (como grupo)_____ el/la docente demuestra físicamente ________ el/la docente demuestra físicamente con los materiales _____ El/la docente dibuja _____ el/la docente utiliza un cartel _____ Los estudiantes los estudiantes leen En voz alta_____ En silencio_____ estudiantes escriben en el pizarrón_____ estudiantes hacen juegos de roles _____ estudiantes hacen demostraciones físicamente_____ estudiantes hacen demostraciones con materiales_____ USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 67 Estudiantes trabajan desde sus asientos_____ otros_____ Técnicas de refuerzo (elija todos los que apliquen): _______________ El/la docente explícitamente agradece a los estudiantes por su participación_____ el docente refuerza con una expresión positiva_____ docente es neutral_____ docente critica a estudiantes individualmente docente critica toda la clase_____ El/la docente se mueve para atender necesidades de disciplina de los estudiantes (sí / no) _____ El/la docente se mueve para atender las necesidades de aprendizaje de los estudiantes (sí/no) _____ Análisis de género: participación de niñas y niños INSTRUCCIONES: Observar durante cinco minutos y contar el número de veces que el/la docente pide a las niñas / niños en la siguiente tabla 1 = las niñas y los niños reciben oportunidades bastante incluso para participar 2 = las niñas reciben más oportunidades de participar 3 = los niños se les da más oportunidades de participar 4 = hay poco a ninguna participación Niñas Niños Observaciones: USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 68 ANNEX H. CONFLICT OF INTEREST DISCLOSURES USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 69 USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 70 USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 71 ANNEX I. EVALUATION SCOPE OF WORK10 Through the U.S. Agency for International Development/Dominican Republic (USAID/DR) Monitoring, Evaluation and Learning (MEL) Platform contract, USAID/DR has engaged Panagora to conduct an external performance evaluation of USAID/DR’s Read activity. Being implemented by the Ibero￾American University (UNIBE), Read is a five-year, $19,999,999 activity which started June 18, 2015 and will end on June 17, 2020. THE READ INTERVENTION The Read activity employs USAID global best practice and experience in the Dominican Republic around engaging communities together with schools to improve early-grade reading. Read supports parents, teachers and students so that each can play their role in improving early-grade reading and some numeracy. Teachers receive mentoring, training, information and communications technology (ICT), materials and tools to enable them to improve their teaching and to support parents. Parents and caretakers receive materials, information and support to help their children study at home and allow their children to attend supplementary reading programs. Students have greater access to materials, access to supplementary reading programs, and individualized support to address any disabilities or special needs which may prevent them from learning effectively. In addition, Read creates an enabling environment within the community through campaigns promoting literacy, safe schools and education for all. These are coupled with incentives that mobilize wider community contributions in the form of youth volunteers, including from groups supported by another USAID/DR activity, Alerta Joven, and the wider community. Campaigns raise awareness in the communities and engage parents to advocate for greater equality in learning for boys and girls. The campaigns employ information generated through community scorecards that provide information on the extent to which the quality of education is improving. In addition to understanding the Read Activity and the expected results, especially with regard to improving early grade reading skills, it is also important to understand that the program is much broader and covers important additional areas. These areas include for example inclusion and inclusive learning environments. In addition, Read covers gender especially gender equity for program beneficiaries (children, teachers and caregivers). Lastly Read covers sustainability and uptake by the government. These aspects of the theory of change and developmental hypothesis are all closely related, and the evaluation will analyze their interconnectedness. PURPOSE AND SCOPE OF THE EVALUATION This performance evaluation is intended to document lessons learned and to inform the design of future education programs which focus on USAID’s Country Development and Cooperation Strategy (CDCS) – Goal 1 (literacy). This evaluation will provide an independent examination of the overall progress and achievements of the Read activity in the Dominican Republic. It will cover Read activities from inception 10 This is the original SOW as submitted to USAID on September 6, 2017. Some minor aspects were modified after the in-brief with the evaluation team. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 72 to date, and will address sustainability and outcomes. The evaluation will identify achievements, performance issues and constraints related to activity implementation. The evaluation will also identify results and lessons learned from implementation and will provide concise, actionable recommendations about which component(s) of Read could be scaled up, modified or re-designed for the final years of implementation and in future procurements in order to enhance overall activity performance. The evaluation findings and recommendations will be shared and discussed with USAID/DR, implementing partners, and MINERD. EVALUATION QUESTIONS USAID identified nine key evaluation questions. These are listed below, along with suggested sub-questions to clarify the intent of the key questions. The evaluation will focus on the key questions, both in the design of instruments and analysis tools, and in the overall findings, conclusions and recommendations of the evaluation report. MANAGEMENT 1. Implementation. What have been the major implementation challenges of the cooperative agreement? a. Has UNIBE responded effectively to these challenges? b. How effective is UNIBE’s coordination with sub-awardees, in particular, World Vision? TECHNICAL APPROACH 2. Reading approaches and strategies. How has UNIBE integrated different reading strategies and resources during teacher training and hence in classroom instruction? a. Has Read implementation (training, resources and materials) been modified in any way as a result of consultations with the MINERD? b. What are the similarities and differences between Read’s reading interventions and reading interventions in schools where only the MINERD is working? 3. Teacher training and use of new reading strategies. Has the training offered by Read helped teachers use effectively the new reading strategies incorporated into the MINERD curriculum? a. Is the training effective in transmitting the new reading strategies to the teachers? b. What aspects of the training are the most useful? What aspects can be improved? c. Have teachers perceived changes in children’s reading skills after applying what they have learned through the Read trainings? What are the main changes? 4. Teacher mentoring. Has the teacher mentoring been effective? a. How frequently does the mentoring occur? Is the frequency appropriate? b. How is the Mentor selection process conducted? Are the Mentors qualified? c. Is the mentoring process effective in improving the quality of reading instruction? If so, how? d. What can be done to improve the teacher mentoring? 5. Safe schools. Has school safety worsened/improved after the Read safe schools intervention? USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 73 a. How is school safety defined and operationalized by UNIBE? b. How does UNIBE assess whether the safe schools and violence-prevention interventions are effective? c. How does UNIBE measure the impact on students of the safe schools and violence￾prevention interventions? Is there a minimum threshold that reflects “success”? d. How effective do teachers consider the Gender, Inclusion and Safe Schools training? Has it been useful? e. Has the safe schools program mentoring follow up been useful? How? Is any change needed? 6. Community involvement. Are the Clubes Leyendo supported by UNIBE and World Vision effective? a. What is the process followed for students to participate in Clubes Leyendo? Is the frequency of club meetings appropriate? Is the time of day appropriate? b. Are parents and the community involved in their children’s learning? Has their involvement increased through the Clubes Leyendo? If so, in what way? c. Has Read been effective in creating and strengthening the Asociaciones de Padres, Madres, Tutores y Amigos de la Escuela (APMAEs)? d. Have parents of students participating in Clubes Leyendo seen any improvement in their children’s reading and writing? If so, in what aspects? SUSTAINABILITY 7. Engagement with MINERD. How successful thus far are Read’s efforts in incorporating significant MINERD engagement and designing a strategy for long-term activity sustainability? a. Are the mechanisms for engagement with the MINERD effective? b. How does UNIBE present the Read activity’s reading “successes” to the MINERD? c. What is the Ministry’s opinion about the effectiveness of this activity to date? 8. Knowledge management. To what extent has Read generated knowledge of effective reading instruction and supportive approaches and shared such information with key stakeholders? a. Has UNIBE developed a comprehensive analysis plan to identify all of the topics and methods of analysis of Read research data? b. Has UNIBE made full use of the data collected in the baseline? c. Do Read’s research reports explain clearly the research questions, research methodology, data limitations, and the findings of the statistical analysis? d. In reporting on research results, does UNIBE explain clearly its methodology (the use of randomized control trials) to measure activity impact? e. Are Read’s research products appropriate for dissemination and discussion with key stakeholders? 9. Recent changes in Read context. How will the MINERD’s new teacher training strategy (with INAFOCAM) affect the Read activity going forward? a. What challenges will the Read project face while working in parallel with another institution within the same schools? b. What will be the impact of the new training strategy on the Read control group? USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 74 10. What would the ministry recommend in order to improve USAID education programming in the Dominican Republic? How could USAID best cooperate with the improvement of reading and quality education in the country? Are there any specific areas, lines of work or activities that the Ministry has identified as areas of potential improvement and is it interested in receiving U.S. government cooperation for these areas? APPROACH The evaluation team will draw on utilization-focused methodologies to ensure that the information generated by the evaluation is useful to USAID. The team will use the initial in-brief meeting to confirm USAID/DR’s goals and objectives and the type of information and insights that will be most useful to USAID’s decision making. The evaluation team will also explore with USAID/DR how the Mission will use the results of this evaluation in the current Read activity and in future programming. GENDER-SENSITIVE PERSPECTIVE Consistent with USAID’s evaluation policy, and recognizing that effects of integration and the success of the activity might vary across gender, the evaluation team will apply a gender perspective to the entire evaluation process. Starting with the background document review, data related to gender-based variances in outcomes will be examined, and activity documents will be reviewed with a gender-sensitive lens, in order to inform a better understanding of gender dynamics in the implementation environment. The evaluation team will include both female and male researchers, and a gender balance among respondents will be sought during the recruitment phase, especially among teachers, as respondents are available. During the data collection process, the evaluation team will endeavor to ensure that a female evaluator is present during interviews with female respondents, and some focus groups will be segregated by gender in order to encourage active participation by all respondents, and to limit the potential for one gender to dominate the conversation at the expense of another. During the data analysis phase, all participant-level data (both qualitative and quantitative) will be disaggregated and reported by sex. This analysis of gender aspects of the Read intervention and discussion of findings, and, where appropriate, findings and conclusions, will be included in the final evaluation reporting (both written and oral). DATA COLLECTION METHODS DOCUMENT REVIEW During the initial phase of the evaluation, the team will conduct a thorough review of background documents related to Read and its implementing environment. These documents will provide the team a deeper understanding of the activity’s operations to date. This will help the team finalize the evaluation design and data collection tools, and will inform the overall activity assessment process, including developing conclusions and recommendations for future programming. These documents will include the following: 1. Cooperative agreement and program modifications 2. Annual Work Plans 3. MEL Plan 4. Annual Reports USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 75 5. Quarterly Reports 6. Baseline Report 7. Special studies • Year 1 end of second grade report • Year 2 end of second grade report • Textbook Bias Study • Report on Attendance 8. Studies conducted by other organizations focused on children/youth in DR 9. Success stories and other relevant communication materials developed for Read 10. Performance Indicator data 11. Site listings and contact matrix 12. Staff contact list 13. List of Read partners and short description of their scope 14. Relevant reports from other donor agencies 15. Relevant GoDR policy documents and technical reports REVIEW OF QUANTITATIVE DATA During the implementation process, UNIBE has collected a wealth of data and produced reports and special studies related to learning outcomes, including the baseline report and first and second year second grade studies. As these data are much more robust and representative of Read’s outcomes to date than anything that could be collected through this evaluation, the evaluation team will work with Read’s quantitative monitoring and evaluation data, as made available by UNIBE. The evaluation team will examine these data to determine the level of progress made by the activity over the implementation period, to map efforts to target children/youth, and to visually display the connections between the type of intervention and performance outcomes. In this regard, the essential data will include (as available): 1. Read monitoring data 2. Read supplementary monitoring data 3. Read baseline survey quantitative data KEY INFORMANT INTERVIEWS The evaluation team proposes to conduct 20-25 KIIs with MINERD, INAFOCAM, local and international NGOs (Dream, Project Hope) and donors (IDB, USAID, OEI), UNIBE and partners (World Vision, Save the Children, Fe y Alegria, and RTI), and private sector partners (for example Fundación Inicia). Data collection protocols that address the evaluation questions will guide KIIs, but will be semi-structured (identifying probes to follow up on information related to key questions) to allow flexibility in the discussion and a natural flow to the conversation, should the evaluation team identify a point of particular interest to the research questions. The evaluation team will conduct field research in 16 Read schools. The team will interview school principals and one teacher per school. Data will be registered electronically. Final selection of respondents will be randomized for larger stakeholder groups, and will include a balance of male and female respondents. In cases where several respondents of the same type are available in the same location and agree to meet together, a group KII or FGD format may be used to USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 76 maximize the number of perspectives gathered during the limited data-gathering period. Based on documents and guidance from implementing partners, the evaluation team will finalize the list of key informants and the sample size for each informant category. FOCUS GROUP DISCUSSIONS The evaluation team will conduct 5 FGDs with parents. The MEL Platform Administrative and Logistics Coordinator will emphasize pre-mobilization to make sure participants will be available and the discussions scheduled at appropriate times. When recruiting FGD participants, the evaluation team will clarify that participants will receive no financial compensation or transportation reimbursement. The evaluation team will use FGDs to obtain the perspectives of 5-7 parents at once, as well as to promote idea sharing and collaborative group assessments. While the exact parties for FGDs will be finalized in conjunction with Read, essential FGDs will include the following. CLASSROOM OBSERVATIONS In addition, the evaluation team will endeavor to undertake one classroom observation to collect qualitative data in order to gain a deeper understanding of the implementing environment and how the curricula are delivered. The evaluation team will work in direct collaboration with the site administrators and facilitators to identify appropriate opportunities for this observation, and will work to minimize disruption of the normal classroom environment as much as possible. Observers will use a rigorous classroom observation tool that reflects what is expected to be seen in the Read approach to guide observations. Observations will not exceed 45 minutes. Each observer will record the observed behaviors in the observation tool during the observation. Then, after checking the tool, the team members will submit the observation tools on a daily basis to facilitate rapid analysis of observation results. Data will be recorded electronically. SITE SELECTION To capture the depth of intervention effectiveness within the given data collection timeframe and team size, the evaluation will focus on visiting Read schools in the Duarte Corridor. The evaluation team will visit 16 school activity sites in relevant areas to understand implementation trends in different locations. DATA ANALYSIS AND REPORTING QUANTITATIVE DATA Analysis of quantitative data made available by UNIBE will naturally depend on how the data have been captured. Ideally, the data will be made available to the evaluation team in a coded file (Excel format or similar), which could be imported into a data analysis software tool and analyzed for standard quantitative measurements such as: frequencies, mean, median, mode, standard deviation, and other inferential statistics as appropriate and relevant to the data analysis. The evaluation team will report quantitative outcomes using tables, charts and graphs in the final evaluation report. They will disaggregate data by sex, learner level, and social demographics for each indicator examined, and comment on findings that vary significantly by demographic cohort. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 77 Qualitative Data The evaluation team will code and analyze the qualitative data gathered through the KIIs and FGDs using systematic qualitative methods. The team will develop a coding system to identify and respond to information sought in the key evaluation questions, and will orient each team member to the coding process to increase inter-rater reliability. In addition, they will code for demographic information, such as gender, age, ethnic background, economic and educational status, etc., which will also be considered during the analysis process and include in the evaluation report. The team will anonymize qualitative data in the evaluation report in order to ensure respondent confidentiality; only the respondent’s sex, age and data collection location will be identified with any specific quotations, in order to provide context for the quotation. General aggregation of qualitative data will be enumerated but not quantified. In other words, the data will be coded and then frequency of codes counted. TEAM COMPOSITION The evaluation team will be composed of four individuals: • Team Leader. Responsible for coordinating the activities of the evaluation team; has the authority to make programmatic decisions regarding the evaluation. She will serve as the main point of contact between USAID and the MEL Platform regarding this evaluation. The Team Leader will draft the final evaluation design, oversee and support the development of evaluation instruments, participate in data collection and analysis, integrate the findings of different team members, and coordinate preparation of the final report. This role will be filled by Megan Gavin. • Evaluation Specialist. Support as needed especially with data collection. In addition, will conceptualize the evaluation instruments, sampling strategy, and data analysis methods per question for the evaluation design matrix, and supports overall analysis. This role will be filled by Marcelo Castillo. • Education Specialist(s). Support data collection, management, and analysis, including report writing. Will carry out school visits, KIIs, and FGDs, and provide expertise and knowledge on the education context in DR, particularly on education programs. This role will be filled by Luis Dominguez. • Logistics Coordinator. Work with local partners to plan travel and schedule data collection, interviews, and assessment activities as required. Will support the Team Leader in all logistical and administrative aspects of the evaluation. This role will be filled by Natalia Folch. The Team Leader is required to have demonstrated expertise in evaluation methodology. Collectively, the evaluation team has experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in DR, all team members will familiarize themselves with USAID’s Evaluation Policy, Gender Policy, publication outlining a good evaluation report, and the checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic policy, and educational context in DR. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 78 The MEL Platform Chief of Party, William Cartier, will be engaged with USAID and the Team Leader throughout to ensure adherence to USAID and MEL Platform quality standards and procedures and to ensure timely completion of the work. DEBRIEF AND REPORT SUBMISSION The team will present initial findings during a debrief at USAID/DR, shortly after the data collection has been completed (Fri. October 27, 2017). The team is also available to present findings to Read (as appropriate and approved by USAID); that debrief would include a discussion of achievements and activities, with a preliminary discussion of recommendations for emphasis on most effective approaches and activities during the final years of implementation. The evaluation team will submit the draft evaluation report to USAID/DR, incorporating feedback provided during the debrief, on Friday November 10, 2017. The report will answer all of the evaluation questions and make clear how each was answered. The team requests that USAID provide comments on the draft report by Friday November 17, 2017. The evaluation team will then revise the draft report to fully reflect USAID comments and suggestions, submitting the final draft by Wednesday November 22, 2017. Panagora will perform final internal quality control to ensure a high-quality product and then submit the final evaluation report to USAID by November 27, 2017. The final report will be reviewed by the TL, CoP and HO to ensure that it meets the criteria laid out in ADS 200 on Criteria to Ensure the Quality of an Evaluation Report. USAID.GOV FINAL REPORT – MID-TERM PERFORMANCE EVALUATION READ ACTIVITY | 79