I M P A C T E V A L U A T I O N of Children’s Resources International, Pakistan (2002-2007) MARCH - 2008 PREPARED BY: Lahore University of Management Sciences (LUMS) Team: • Dr. Muhammad Farooq Naseer • Ms. Manasa Patnam • Dr. Reehana Raza • Mr. Muhammad Yasir Khan • Mr. Muhammad Asim Aftab Page | III IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) TABLE OF CONTENTS CONTENTS II ACKNOWLEDGMENTS III LIST OF ACRONYMS IV LIST OF FIGURES, GRAPHS AND TABLES V EXECUTIVE SUMMARY VI 1. INTRODUCTION 1 2. THE CRI PROGRAM 3 3. METHODOLOGY 5 3.1. Sampling Issues 5 3.2. Propensity Score Matching 7 3.3. Testing and Survey 8 3.4. Data for the Board Exams 9 4. THE EVALUATION RESULTS 10 4.1. The Success of the PSM 10 4.2. Learning Achievements 12 4.3. Attendance 14 4.4. Gender and Wealth 15 4.5. Board Results of Grade V, 2006 - 07 17 4.6. Sensitivity Analysis 18 5. CONCLUSION 20 REFERENCES 22 APPENDIX 23 A. Teachers’ Questionnaire B. Child Questionnaire Page | V IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) ACKNOWLEDGMENTS The LUMS Team would like to take this opportunity to thank a number of people without whom this evaluation would not have been possible. Firstly, we would like to thank Professor Rafi que Tahir, the Director of the Federal Directorate of Education, and his staff for providing access to schools in Islamabad and offering guidance on the study. We would also like to thank Mehnaz Aziz, Chief Executive of Children’s Resources International Pakistan (CRI-P) and her staff for their support and time for this evaluation. Various academicians supported this endeavor and offered comments on this study for which we are grateful. Included among them are Tahir Andrabi (Pomona College), Faisal Bari (LUMS), Ali Cheema (LUMS), Ehsan Choudhri (Carleton University), Jishnu Das (World Bank) and Asim Khawaja (Harvard). RCons, the survey organization contracted to undertake the testing and led by Irfan Ahmad who very ably and professionally undertook the testing exercise. We would also like to acknowledge the help of our research assistants, Muhammad Yasir Khan and Muhammad Asim Aftab. Finally, we thank the principals, teachers and students of the schools included in our study. Page | VI IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) LIST OF ACRONYMS ATT Average Treatment Effect on the Treated CRI Children’s Resources International Pakistan EMIS Education Management Information Systems FDE Federal Directorate of Education ICT Islamabad Capital Territory IECRI Impact Evaluation of Children’s Resources International Pakistan IMC Islamabad Model College IRT Item Response Theory LEAPS Learning and Educational Achievement in Punjab Schools ITLP Interactive Teaching and Learning Program LUMS Lahore University of Management Sciences MTTs Master Teacher Trainers NGO Non-Governmental Organization PSM Propensity Score Matching PPP Public-Private Partnership RCons Research Consultants, Inc. USAID United State Agency for International Development Page | VII IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) LIST OF FIGURES, GRAPHS AND TABLES FIGURES Fig. 1: School District Map 04 Fig. 2: Stratifi cation of Blocks 07 Fig. 3: Mean Age by Treatment and Child Blocks 12 GRAPHS Graph 1: Types of Schools in Different Sectors 05 Graph 2: Schools by Medium of Instruction 06 Graph 3: Distribution of Scores before Controlling for Child Characteristics 12 Graph 4: Distribution of Board Results without Controlling for Child Characteristics 17 TABLES Table 1(a): Content Areas for Urdu and English 08 Table 1(b): Content Areas for Mathematics 09 Table 2: Descriptive Statistics before matching 10 Table 3: ATT: Full Sample IRT Scores (uncontaminated) 13 Table 4: ATT: Restricted Sample IRT Scores (uncontaminated) 14 Table 5: ATT by Gender (uncontaminated) 15 Table 6: ATT by Wealth (uncontaminated) 16 Table 7: Descriptive Statistics: Board Results 17 Table 8: ATT for Board Results 18 Page | VIII IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) EXECUTIVE SUMMARY CRI introduced its pedagogy in 25 public sector primary schools in Islamabad in 2002 under Creating Democratic Schools1 Program with the funding support of USAID. By 2007, the fi rst cohort of pupils who had their complete primary education under the CRI pedagogy fi nished their schooling. What impact has this child-centered instructional approach had on the learning outcomes of these students? This impact evaluation compares the learning outcomes of children in the 25 piloted CRI partner schools with other school children that are similar in all respects except exposure to the CRI program. The study is quantitatively rigorous, as it draws upon a survey of 2000 CRI and Non-CRI students and controls for a number of confounding factors to estimate the CRI effect. The evaluation used a testing instrument developed for The World Bank funded Learning and Education Achievement in Punjab Schools (LEAPS) project. Based on this comparison it was found that the CRI program has been effective in raising learning achievement. The average cumulative gain for a CRI student with four years of exposure to the program represented an improvement of 0.23 standard deviations and an increase in the ranking by 4-11% above her current standing vis-à-vis other students in her cohort. In terms of the learning achievement, we found a positive and highly signifi cant effect of the CRI program. Being from a CRI partner school improved a student’s overall performance on the test by 5.6 points (a difference of 3.9 percentage points in the total score). That is, a typical student exposed to CRI would achieve 5.6 more points as compared to a representative student from a Non-CRI school. Among the three subjects tested, the effect was largest in magnitude for English and Urdu as opposed to Mathematics where the results were marginally favorable for the students in CRI partner classrooms. The estimated effect on the attendance rate was also found to be positive as average attendance in the last three months was 4.2 percentage points higher for CRI children as compared to matching children in Non-CRI schools. Disaggregation of results by gender and wealth indicated a slight advantage in the favor of boys relative to girls and a larger advantage for children of higher income households. Our research also established that the causal relationship between the CRI program and learning was robust to the presence of unmeasured confounding variables. Finally, the study also tests whether there was any signifi cant difference in performance between the CRI and Non-CRI school in the external board exams 2006-2007 for Grade V. Based on the aggregate board marks, and controlling for socio-economic heterogeneity, we fi nd that students in CRI partner classrooms perform no differently relative to the Non-CRI students on Grade V board exams. What explains these seemingly contradictory results on the LEAPS instrument and Grade V board exams? Why do students in CRI partner classrooms not outperform Non-CRI students on the Grade V exam? The most likely explanation is that Grade V board exams refl ect students’ preparedness for the exam while the LEAPS instrument captures student ability at a given point in time without giving them the opportunity to prepare. Taken together the results suggest that CRI exposure improves learning but does not negatively affect performance on the board exams. 1 Creating Democratic Schools (CDS) Program was implemented by CRI with the funding support of USAID from 2002 to 2005. This was subsequently followed by the Interactive Teaching & Learning Program (ITLP) in Pakistan which commenced in 2006 with the funding sup￾port of USAID. Page | 1 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 1. INTRODUCTION Classroom innovations are rare in public sector schools in Pakistan. An extensive system of formalized examinations and cultural norms that value hierarchy ensure that classroom teaching remains didactic. Classes are teacher-led, have limited teacher child interaction, and are focused on maximizing skill transfer--often through rote learning, with little exploration outside the government determined curriculum. Limited resources mean that few initiatives are taken to visualize concepts or experiment with tools and little room is given for children to explore ideas outside the realm of their textbook. Teachers have limited incentives to experiment in classrooms and often do not have the necessary skills. There are also broader challenges facing the school administers which make issues of instructional approach almost secondary. This includes lack of teachers, lack of physical infrastructure and often a lack of students. One recent exception has been the introduction of a child-centered instructional approach in primary schools in the public sector in Islamabad Capital Territory (ICT) through a Public Private Partnership (PPP) between the Federal Directorate of Education2 (FDE) and Children’s Resources International Pakistan (CRI), a non-profi t organization that is engaged in improving the quality of education in Pakistan. CRI’s focus is to democratize the learning environment, encourage children to take the initiative and make active choices and through this process encourages them to be independent and responsible individuals. In 2002, CRI began working in 25 public schools in ICT as a pilot program under “Creating Democratic Schools3 Program” with the funding support of United State Agency for International Development (USAID). The entering cohort of the kindergarten students in 2002 in these pilot schools has recently completed their whole primary education under the CRI approach. From 2006 the CRI program entitled “Interactive Teaching and Learning Program” in Pakistan with the funding support of USAID extended to 300 plus schools in Islamabad schools, essentially covering all public schools involved in teaching grades kindergarten to Class 8 in the country’s capital. The CRI intervention is essentially a package of mixed educational inputs which includes teacher training, family literacy and the development of school facilities as well as supplying certain school equipment. The package is introduced into existing government run primary schools under existing management structures. Preliminary anecdotal evidence from the CRI pilot schools suggests that the CRI intervention has had a real impact with improvements in enrollment levels and retention rates along with a decline in drop-outs. Children’s Resources International recognizes that these results are only indicative and not conclusive about the impact of the CRI program. A more comprehensive evaluation of the CRI intervention is thus timely. First there is a need to offer a rigorous evaluation of the Program’s impact in terms of their learning achievement. Second the schools introduced to the intervention in the pilot phase were to produce their fi rst cohort of students in 2007 that would have completed their entire elementary education under the CRI method.4 In light of this CRI approached the Department of Economics at Lahore University of Management Sciences (LUMS) to collaborate on developing a methodology for conducting the impact evaluation of the CRI program in Phase Zero. The objective of the methodology is to measure the impact and assign causality to the CRI Program against relevant outcomes, in this case learning achievement in Urdu, English and Math as well as attendance. The Phase Zero evaluation is in two parts: the fi rst part assesses learning outcomes of CRI vs. Non-CRI students based on their performance on a test administered by us in Grade IV of sample schools and 2 The Federal Directorate of Education is a department of the Ministry of Education responsible for the implementation of education policy in Islamabad. Pakistan is divided into four provinces, two centrally administered areas and a capital territory. In each of these areas, education policy is administered by the provincial or equivalent authority as per the Pakistani constitution. 3 Creating Democratic Schools (CDS) was implemented by CRI with the funding support of USAID from 2002 to 2005. This was subsequently followed by Interactive Teaching & Learning Program (ITLP) in Pakistan which commenced in 2006 with the funding support of USAID. 4 For the purpose of this report, the initial pilot stage of the Program in 25 schools is called the Phase Zero of the Program. Page | 2 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) the second part compares CRI and Non-CRI students on the Grade V Board exams. The methodology for the Grade IV assessment was developed in January-March 2007. Survey and testing of students was conducted in April 2007 with the assistance of the RCons Consulting Group. Data analysis and report writing for the fi rst part of this evaluation was completed in September 2007. The analysis of the board results was completed in December 2007. Page | 3 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 2. THE CRI PROGRAM The overall goal of CRI Pakistan’s program is to promote high quality education and critical thinking skills. The CRI Program is designed to improve pedagogy and student performance through teacher training, faculty development and family literacy. It adopts a teaching style which shifts learning from being teacher-centered to student-centered. The philosophy of this program is rooted in the belief that children develop best when they are intrinsically involved in their own learning. CRI methodology encourages children to make choices and take responsibility for their decisions, express their ideas creatively and develop critical as well as independent thinking skills. CRI facilitates teachers in setting up activity centers to help children interactively explore their lessons. The centers include Mathematics, Science, Literacy, Art and Dramatic Play and are set up using the learning material provided by CRI. Each center attempts to clarify concepts introduced in the curriculum. In addition, children’s work is displayed on the wall and portfolios introduced for an on-going assessment of children. Learning is made fun for children through the Morning Meeting. This provides choice time for children to share books, celebrate hundred days, math day and science day and through dramatics. In the Morning Meeting, children sit in a circle, greet each other, and make eye contact. A sense of responsibility and respect for each other is encouraged and children share small things. During Choice Time, different choices are given to the children in the schedule to take initiative and make learning interesting and pleasurable. In Book Making and Author’s Chair time, Children make their own story books and share it with their class-fellows from the Author’s Chair. On Hundred Days in school, the children count the number of days they have attended school; when hundred, they celebrate that achievement in different ways. On Science Day, the little scientists are encouraged to undertake scientifi c projects while on the Math Day, different activities are designed to integrate mathematics with day-to-day life. In Adopt-a-Plant activity; children plant a tree, take ownership of its care and record monthly changes. At Dramatic Play, children assume different roles and make props and talk to each other. These activities, in different ways, promote a child’s cognitive and social development. The CRI Program also emphasizes regular teacher training. CRI believes that currently taught methodologies rely heavily on the teacher transferring knowledge to students through memorization rather than facilitating the students to learn on their own. Seeing this, the NGO has developed comprehensive specialized faculty courses on Basic Education. These courses are designed to strengthen and enhance the skills, knowledge, and effectiveness of faculty members involved in the fi eld of education and are in use in more than 300 international institutions of higher education that train teachers. The guidance that teachers receive allows them to implement active learning by setting up different activity centers in their classrooms. Each teacher’s individual progress is monitored by the Master Teacher Trainers (MTTs), who regularly make follow-up visits to schools in order to observe classrooms and provide hands-on guidance where necessary. Lastly, the Parent and Community Involvement Program seeks to involve parents in school and classroom through various planned activities. This program is based on the premise that the more the parents know about their children’s education, the more apt they are to extend the learning at school into their home life. In CRI partner schools, families are encouraged to participate in school and classroom activities and schools designate Family Coordinators to promote parental involvement. The said Program is implemented by inviting parents to the school on a regular basis and involving them in school activities. Sometimes, the parents are invited to come into their children’s classes to assist the teachers by telling stories, sharing their experiences or by demonstrating their professional skills. The Parent and Community Involvement Program enlists the aid of Page | 4 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) parents in educating their children. It also gives parents a sense of achievement, opens communication channels between parents and teachers and creates opportunities for parents to help their children do better in school. FIGURE 1: SCHOOL DISTRICT MAP The above fi gure shows the growth of the CRI Program over three phases from 2002 to 2007. Since 2002, CRI has been working in 25 government schools in Islamabad*5 . For the purpose of this evaluation, the initial intervention in the pilot schools is referred to as Phase 0. In Phase 0 schools, the CRI program was introduced in yearly steps and by August 2006 all grades from KG to 5 were introduced to the CRI methodology. Then CRI and the FDE (Federal Directorate of Education) reached an agreement in 2005 to extend its work to all the government schools in Islamabad. As a result, by 2006, 141 more schools came under the CRI Program; this particular phase of the Program is referred to as phase 1. During 2007 and 2008, the CRI program was to be introduced to the remaining government schools with the potential to impact 100,000 students, over 3,000 classrooms (2006-2009) and 5000 parents. 5 CRI has been working in three areas i.e. Islamabad Captial Territory (ICT), Rawalpindi and Karachi. However, this research was conducted only in ICT’s schools of Phase - 0. Page | 5 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 3. METHODOLOGY The primary focus of this evaluation has been to assess the impact of the CRI intervention on learning achievement and attendance. Learning outcome is perhaps one of the most signifi cant outputs of the schooling process as it signals the quality of education delivery and has a long-term cumulative effect on a student’s academic performance. To evaluate the impact of CRI intervention, the focus was on measuring differences in student achievement between schools which received the CRI intervention, vis-à-vis those that did not. A serious challenge to devising this strategy was the absence of any baseline study at the time of initiation of the CRI intervention. Further it could not be assumed that the selection of schools for the purpose of the CRI intervention was done randomly. 3.1. Sampling Issues Twenty fi ve CRI partner schools in Islamabad were included in the treatment group. Forty four non CRI schools were selected under the PSM methodology as control group. In total 69 CRI and non CRI partner schools were selected. Tests of English, Urdu and Maths were conducted among 1984 students of these schools. 741 students were from CRI Partner’s Schools and 1243 were from non CRI schools. The driving force behind the sampling strategy was to address the twin issues of a lack of baseline data and the non-randomized selection of pilot schools. No data at the onset was collected on household characteristics and neither was there any baseline data on students’ learning achievement, which made it impossible to look into the effect of CRI intervention on these children by comparing them with their own baseline performance. One solution to this problem was to compare CRI sponsored schools with Non-CRI schools. But, as explained below, schools in Islamabad show signifi cant variation in their observed characteristics even if compared within residential sectors, thus making the demarcation of a comparison group (consisting of Non-CRI Schools) extremely important. The following graphs show the range of variation in the type of schools found in different sectors of Islamabad.6 GRAPH 1: TYPES OF SCHOOLS IN DIFFERENT SECTORS 0 5 10 15 20 25 30 35 40 45 50 EFGH I Sector Number of Schools Islamabad Model College/School Primary/Middle School Junior Model School Source: FDE EMIS Database; Islamabad School Census 2002-03 6 There is signifi cant socio-economic heterogeneity between different sectors of Islamabad. The government run schools in each sector cater to the needs of the local residents. Page | 6 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) The graph highlights the different types of schools as well as the number of schools within ICT. Islamabad has three different types of public schools which vary signifi cantly in quality and characteristics of entering students. At the top of this hierarchy sit Islamabad Model Colleges (IMC) which, in some cases, can go from kindergarten to Masters level degrees. These schools have much better infrastructure and their teaching faculty is required to possess a Bachelors degree at a minimum. These model colleges and schools are regarded as the best quality institutions in the public sector. Next in the hierarchy are the Junior Model Schools. In general, they have less qualifi ed teachers than IMCs, as the minimum qualifi cation for a teacher in these schools is the completion of grade 9.7 A feature that distinguishes them from the simple Primary Schools at the bottom of this schooling ladder is that these were English medium institutions at the time of school selection in 2002. The Primary Schools, on the other hand, at time of selection for the CRI Program were Urdu-medium schools8 , which are not co-educational and the teaching faculty is required to be all female. In Pakistan, the medium of instruction is often an important predictor of school quality. It is a common perception that children from schools where the medium of instruction is English normally do well as compared to children from Urdu-medium schools. The following graph shows the distribution of schools by medium of instruction in the different sectors of Islamabad. We can see that there is a lot of variation in the medium of instructions within the different sectors. GRAPH 2: SCHOOLS BY MEDIUM OF INSTRUCTION 0 10 20 30 40 50 60 EFGH I Sector Number of Schools Urdu Medium English Medium Source: FDE EMIS Database; Islamabad School Census 2002-03 In addition, not all sectors are homogenous in terms of the socio-economic condition of the residents. Thus, it is not surprising to fi nd the above variation in the distribution of public schools across different urban residential sectors in Islamabad. The distribution of schools by medium of instruction vis-à-vis the socio-economic characteristics do not always correlate as we would expect. For instance, sector E is generally home to well-to-do families but has only a few government run schools and all of them are Urdu medium. This refl ects the fact that most well-to-do families send their off springs to private schools. These public schools may cater to the needs of domestic servants 7 Source: FDE EMIS Database 8 The FDE is currently implementing a plan of converting the Urdu-medium primary schools to English-medium Junior Model Schools. Page | 7 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) working in that sector. Sector G on the other hand has a broad socio-economic structure and consequently has a mix of schools of English and Urdu medium. Therefore, it would not be appropriate to naïvely compare schools without taking into account these important school-sector differences. 3.2. Propensity Score Matching In light of the above, we use the method of propensity score matching (PSM) to select a comparable sample of the Non-CRI schools. The basic idea behind PSM is to compare two subjects having the same likelihood of receiving the intervention despite the fact that only one has actually been selected for treatment. The following fi gure simplifi es the explanation. As the diagram shows, the objective under PSM is to identify subjects in the untreated group who have the same characteristics as the treated group except for treatment. Each subject is thus assigned a probability of being selected for the treatment. Subsequently the subjects in the two groups (treated and untreated) who have the same predicted probability for selection into the set of pilot schools are compared to determine the impact of the intervention. We utilized a two stage matching procedure. In the fi rst stage, schools were matched from all over Islamabad based on school characteristics obtained from the Islamabad school census (2003). In particular, the FDE chose to select schools for this program on the basis of several criteria; geography, infrastructure of schools, medium of instruction and school type. Based on these characteristics, we fi rst drew a sample of matching Non-CRI schools for comparison with the pilot CRI schools. In the second stage, students within these sample schools were matched based on student, household and teacher attributes. This step was necessary because after having stratifi ed CRI and Non-CRI schools into matching blocks, any difference in test scores could still be attributed to individual and household characteristics of the child, such as age, education and wealth of parents and number of siblings as well as the experience and qualifi cation of his or her teacher. Therefore, each school block was further divided into matching sub-blocks to balance the above variables across CRI and Non-CRI students, resulting in children in each sub-block with similar characteristics. Figure 2 presents a graphical exposition of the two-stage matching process by showing the creation of school and child-blocks. FIGURE 2: STRATIFICATION OF BLOCKS Note:” T” represents treatment, “C” represents control. Value in parenthesis denotes the range of probabilities in each block. Page | 8 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) After dividing the children into different blocks, we compared the tests scores of CRI and Non-CRI children using a weighted average estimator. We refer to this estimator as Average Treatment Effect on the Treated (ATT). 3.3. Testing and Survey A key element of ensuring the quality of the evaluation was to choose a testing instrument appropriate for grade IV curriculum being followed in Islamabad schools. The objective was to gauge students on their ability in English, Urdu and Mathematics and to measure if students in CRI Partner School performed different from students not in Non-CRI Schools. The decision was made to use a standardized testing instrument developed by the Learning and Educational Achievement in Punjab Schools (LEAPS) Project.9 The content of these tests is comprehensive and has been developed keeping in mind various aspects of the curriculum of primary schools in Pakistan. In order to ensure the applicability of the test to Islamabad, the test was cross-checked against the Islamabad curriculum and textbooks. The English and Urdu sections had questions on reading comprehension, sentence completion, grammar and vocabulary. The Math section tested children on their mastery over various arithmetic operations, word problems and LCM etc. The testing instrument contained questions with a varying range of diffi culty. The different types of questions included in Urdu and English test are given in Table 1(a) and the ones for Math are listed in Table 1(b). The English test had questions with a diffi culty level ranging from -6 to 3, so that it could test children over a wide range of knowledge. Mathematics test had a diffi culty range of -5 to 3 with one question having a very high diffi culty of 14. Urdu on the other hand a well dispersed diffi culty range of -5 to 3.10 TABLE 1(A): CONTENT AREAS FOR URDU AND ENGLISH IN THE LEAPS TEST 9 The LEAPS Project is a multi-year study funded by the World Bank which is focused on evaluating learning achievement in both public and private schools in three districts of Punjab. For further details, please see Andrabi, T., Jishnu Das and Asim Khwaja, “Test Feasibility Survey Pakistan: Education Sector”, 2002. 10 The diffi culty level of a question denotes the knowledge or ‘ability’ required to answer that question with a 50% chance of getting it right (assuming zero probability of getting it right by a guess). A large positive value for the diffi culty parameter indicates few correct answers to that question except by the most capable students. These diffi culty parameters for each question were estimated by using a standard 3-parameter item response model. Page | 9 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) TABLE 1(B): CONTENT AREAS OF MATHEMATICS IN THE LEAPS TEST Source: Test feasibility survey Pakistan: Education Sector, by Tahir Andarabi, Jishnu Das and Asim Ijaz Khwaja 2002 Along with the tests, a child questionnaire was also implemented in the sample schools to control for the household and child characteristics in the analysis (i.e. in the second-stage matching). Similarly a teacher questionnaire was administered to the class teacher of the students who underwent these tests. This enables us to control for factors like the experience and qualifi cation of teacher. 11 3.4. Data for the Board Exams Besides testing the grade IV students, we also obtained child and teacher information for the grade V students in our sample schools. These students later appeared on the board exams in May 2007. We were able to access the exam results through the Federal Directorate of Education in the form of a data fi le which contained total marks received by each student in the board exam. In order to examine the effect of CRI Program on board exam performance, we employed the same strategy as we had previously adopted for measuring the differences in test scores of Grade IV children (PSM). 11 The teacher and child questionnaires are given in the appendix to this document. Page | 10 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 4. THE EVALUATION RESULTS The objective of the evaluation was to assess the impact of the CRI intervention in the initial pilot schools (Phase Zero) by appropriately assigning causality to the CRI intervention for learning outcomes. Students in the CRI Partner schools who were selected to be tested at the time of testing had the possibility of being exposed to the CRI child-centered approach for a maximum of fi ve years. The results therefore refl ect the cumulative impact of exposure to the CRI Program and focus on both the gains in learning achievement as well as attendance in schools. 4.1. The Success of the PSM Table 2 presents the descriptive statistics for CRI and Non-CRI children in our sample. The table pools together the data from all school and child blocks and reports the mean characteristics for CRI and Non-CRI group in the full sample, prior to matching. The last two columns in the table report the difference in the mean characteristics between the two groups along with its standard error (asterisks denote statistical signifi cance). In the full un-matched sample, there are several signifi cant differences between CRI and Non-CRI subjects both in terms of school and individual/household attributes. On average, we fi nd that CRI children in our sample are younger in age, come from wealthier families with professional fathers, and have fewer siblings than Non-CRI children. Moreover, CRI kids also tend to be slightly smaller in height and weight as compared to Non-CRI children. In addition, relative to the Non-CRI children, a higher proportion of the CRI children tend to come from the English-medium FG Model schools with better qualifi ed teachers. It is clear that the two groups of children differ on a whole range of characteristics apart from their exposure to the CRI Program. Hence, a simple difference in test scores would confl ate the impact of the CRI Program with the net effect of all these other characteristics. The division of schools and children, CRI as well as Non-CRI, in matching blocks with similar characteristics helped overcome this problem by controlling for the difference in these other characteristics. TABLE 2: DESCRIPTIVE STATISTICS BEFORE MATCHING Variable CRI Non-CRI Diff SE Mean SD Mean SD Mean CHILD Male† 0.53 0.50 0.56 0.50 -0.03 0.02 Age in years 9.91 1.26 10.46 1.61 -0.56** 0.07 Mother went to school† 0.23 0.42 0.24 0.42 -0.10 0.02 Father is professional† 0.61 0.49 0.53 0.50 0.07** 0.02 Father is entrepreneur† 0.15 0.36 0.17 0.38 -0.02 0.02 Number of siblings 3.42 1.79 3.81 1.95 -0.39** 0.09 Number of older siblings 1.74 1.59 1.86 1.72 -0.13 0.08 Asset index 0.25 2.19 -0.14 2.22 0.40** 0.10 Child height 137.74 8.73 139.09 10.02 -1.35* 0.44 Child weight 63.46 16.48 65.88 17.80 -2.42* 0.80 Page | 11 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Variable CRI Non-CRI Diff Mean SD Mean SD Mean SE TEACHER Teacher academic qual.† 0.97 0.17 0.87 0.33 0.09** 0.01 (FA/FSc or above) Teacher experience (yrs) 8.62 7.28 8.02 7.28 0.60 0.33 SCHOOL English medium† 0.87 0.33 0.57 0.49 0.30** 0.02 Isl. Model College† 0.24 0.43 0.22 0.42 0.02 0.02 F.G. Model School† 0.63 0.48 0.49 0.50 0.14** 0.02 Boys school† 0.10 0.30 0.22 0.41 -0.12** 0.02 Co-educational† 0.50 0.50 0.49 0.50 0.01 0.02 Urdu score 24.26 8.14 21.48 8.23 2.78* 0.38 Math score 21.36 6.42 19.99 6.81 1.37* 0.31 English score 30.81 7.49 27.44 8.53 3.37* 0.38 Total score 76.43 19.65 68.91 21.58 7.53* 0.97 Urdu score (IRT) 26.55 5.68 24.24 6.00 2.31* 0.28 Math score (IRT) 24.21 4.01 22.55 4.28 1.65** 0.19 English score (IRT) 31.39 5.97 28.96 6.31 2.43** 0.29 Total score (IRT) 82.14 15.62 75.76 16.59 6.39* 0.75 Average attendance 0.86 0.09 0.88 0.08 -0.02** 0.00 Number of Observations 741 1243 Source: Authors’ calculation. IECRI survey † indicates dummy variable; **indicates signifi cance at 1% level. Note: This simple comparison of means is done after pooling observations from all the different blocks together. Thus, as expected, CRI and Non-CRI children are not evenly matched along several dimensions as indicated by signifi cant differences in the mean values of covariates The success of PSM can be seen in Figure 3 where the mean age of CRI and Non-CRI students is compared within child sub-blocks. Although the two graphs do not exactly overlay each other, the average age within a block is quite similar (except for a few outliers).12 12 The dissimilarity in those blocks is due to the fact that we are really matching children on the basis of multiple characteristics. A regression￾based evidence of this multivariate balancing of characteristics within blocks is also available upon request. Page | 12 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) FIGURE 3: MEAN AGE BY TREATMENT AND CHILD BLOCKS Source: Authors’ calculation, IECRI survey 4.2. Learning Achievements A preliminary look at the data indicates that the students in CRI partner classrooms did well on the test relative to the Non-CRI children. Graph 3 plots the distribution of test scores without controlling for child characteristics. The scores of CRI children are distributed to the right indicating their overall better performance on the test. GRAPH 3: DISTRIBUTION OF SCORES BEFORE CONTROLLING FOR CHILD CHARACTERISTICS Source: Authors’ Calculation IECRI survey Page | 13 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) In order to control for individual heterogeneity, subject wise test scores and aggregate test scores were calculated and compared across the children in similar blocks. Moreover, since, un-weighted aggregate score obtained by each student may not be a good measure of a test-taker’s (unobserved) ability in the presence of random guessing by examinees and the inclusion of questions with varying diffi culty levels in the test, IRT-adjusted test scores were used to compute treatment effects. More specifi cally, a three parameter logistic item response model was fi tted to the test data and estimates from the model were used to construct a true score for each child defi ned as the expected test score on each section of the test. Additionally, the sample included some CRI partner schools where teachers without CRI training were found as well as a few Non-CRI schools having CRI￾trained. The assignment of teachers to government schools in Islamabad is done by the Federal Directorate of Education (FDE) and transfers across schools also happen at the FDE’s discretion. During the pilot phase of the CRI Program, 2002-06, there was an explicit understanding between CRI and FDE that no CRI trained staff will be transferred from these pilot schools. This reassignment of teachers between CRI and Non-CRI schools represents a recent event that has occurred with the expansion of the program to all Islamabad schools in 2006. So, these teachers could have been transferred at any time since the end of summer 2006. Since, this contamination of the treatment and comparison group would have biased the estimates. All such observations (less than 18% of the full sample) were dropped. TABLE 3: AVERAGE TREATMENT EFFECT ON THE TREATED: FULL SAMPLE IRT SCORES (UNCONTAMINATED) Treatment Eff ect (atts) T-stat 95% Confi dence Interval Urdu 2.00** 3.957 0.985 3.016 Math 1.421** 4.623 .803 2.040 English 2.131** 4.528 1.185 3.076 Total score 5.553** 4.277 2.944 8.161 Avg. Attendance 0.042* 2.189 .003 .081 Treatment Control Total No. of observations 490 298 788 Source: Authors’ Calculation, IECRI survey ** indicates signifi cance at 1% level * indicates signifi cance at 5% level Table 3 reports the effect of CRI on IRT-adjusted test scores after dropping all those students with re-assigned teachers. In the resulting sample, there were 490 children in the treatment (CRI) schools who were matched with 298 children in the comparison schools. The test score all show a positive and highly signifi cant effect of the CRI program, with the effect being largest in magnitude for English and Urdu. The effect of the CRI program on a student’s overall performance on the test is to increase the average score by 5.6 points. That is, a student exposed to CRI will be a priori expected to correctly answer 5.6 more questions as compared to a similar student in a Non-CRI school. An effect of this magnitude on student test scores is quite big. One has to keep in mind that while the actual size of the learning effect varies from individual to individual, the above estimates capture the average CRI effect on student’s learning. A difference in expected IRT-adjusted score of 3.0 (say) implies an increase in the student’s rank by about 5 percentiles at the mean of student score distribution. Page | 14 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) In order to remove potential bias from the cumulative effect of the CRI instructional approach, the sample was divided into those who had been in the said school (both CRI and non CRI) from kindergarten or grade 1 and those who had entered later into the program (grade 2, 3). Subsequently, Tests scores were examined for the restricted sample, i.e. those students who had been in their present school for at least four years. TABLE 4: AVERAGE TREATMENT EFFECT ON THE TREATED: RESTRICTED SAMPLE IRT SCORES (UNCONTAMINATED) Treatment Eff ect (atts) T-stat 95% Confi dence Interval Urdu 1.376 * 2.584 0.305 2.446 Math 0.987 * 2.393 0.158 1.816 English 1.462 * 2.472 0.273 2.650 Total score 3.826 * 2.485 0.732 6.920 Avg. Attendance 0.011 1.031 -0.010 0.032 Treatment Control Total No. of observations 340 160 500 Source: Authors’ Calculation, IECRI survey * indicates signifi cance at 5% level Table 4 reports the average treatment effect obtained when this restricted sample was stratifi ed into blocks of similar CRI and Non-CRI children using child and teacher quality variables, as before. The difference in all subject test scores is now signifi cantly positive at 5% level, that is, CRI children with a minimum four years of exposure perform better than similar Non-CRI children in each component of the test. The cumulative difference between CRI and Non-CRI is 3.8 points. The point estimates are lower than the estimates obtained using the full sample in Table 3, which implies that the earlier results were biased upwards perhaps due to self￾selection by more able children into CRI partner schools (net of the lower cumulative effect of CRI on these late entrants). 4.3. Attendance The CRI program gives great importance to child-centered classrooms while putting emphasis on making the learning environment interesting and fun for children. Does exposure to CRI have a positive effect on a child’s school attendance? The attendance rates over last three months prior to survey are reported in table 3. Attendance has been higher by 4.2 percentage points for CRI children as compared to Non-CRI children, a difference that is statistically signifi cant at 5% level. We however do not obtain the same result in the restricted sample when we consider only those students who have stayed in their present school since grade I. The difference in the average attendance rate between CRI and Non-CRI students is not statistically signifi cant in Table 4. So overall, the results on attendance are not conclusive. Page | 15 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 4.4. Gender and Wealth Pakistan has low female participation in education and literacy levels for females are low. Does a pedagogical approach that is nondidactic favor girl children in the classroom? Performance by wealth can also be revealing. Some of the USA literature on non-didactic learning indicates that children of poor backgrounds perform better under this pedagogy. Thus, a comparison of performance by economic background would be interesting given that home environment and parental resources likely interact with the classroom environment to determine child learning outcomes. In order to undertake this analysis, we use data from the uncontaminated restricted sample and disaggregate it by gender and wealth to obtain separate impact estimates (ATT) after defi ning new child level blocks. TABLE 5: ATT BY GENDER (UNCONTAMINATED) A. Girls Treatment Eff ect (atts) T-stat 90% Confi dence Interval Urdu 2.253 1.494 -0.276 4.782 Math 1.616+ 1.696 0.185 3.213 English 2.395+ 1.951 0.336 4.453 Total score 6.264+ 1.841 0.558 11.96 Avg. Attendance 0.022 1.661 -0.000 0.043 Treatment Control Total No. of observations 180 82 262 B. Boys Treatment Eff ect (atts) T-stat 90% Confi dence Interval Urdu 2.410** 3.633 1.297 3.521 Math 1.716** 3.346 0.855 2.575 English 2.554** 3.133 1.187 3.919 Total score 6.679** 3.365 3.351 10.006 Avg. Attendance -0.027+ -1.689 -0.054 -0.000 Treatment Control Total No. of observations 129 223 352 Source: Authors’ Calculation, IECRI survey ** indicates signifi cance at 1% level * indicates signifi cance at 5% level + indicates signifi cance at 10% level Page | 16 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Table 5 reports the estimated CRI effect by gender. Interestingly, the point estimates for the program’s impact on boys and girls are very close for the learning outcomes and slightly higher for boys. However, girls attend school more often in the CRI partner schools (which is signifi cant at 11%) as opposed to a reduced attendance rate for boys. TABLE 6: ATT BY WEALTH (UNCONTAMINATED) A. Top 35% of Wealth Distribution Treatment Eff ect (atts) T-stat 90% Confi dence Interval Urdu 1.977** 2.912 0.83 3.115 Math 1.434** 3.678 0.780 2.088 English 2.090* 2.312 0.575 3.605 Total score 5.501** 3.024 2.451 8.551 Avg. Attendance 0.005 0.309 -0.023 0.033 Treatment Control Total No. of observations 149 123 272 B. Bottom 35% of Wealth Distribution Treatment Eff ect (atts) T-stat 90% Confi dence Interval Urdu 1.383 1.107 -0.712 3.479 Math 0.948 1.105 -0.491 2.387 English 1.501 0.950 -1.148 4.150 Total score 3.832 1.126 -1.875 9.540 Avg. Attendance -0.023 -0.759 -0.075 0.028 Treatment Control Total No. of observations 81 72 153 Source: Authors’ Calculation, IECRI survey ** indicates signifi cance at 1% level * indicates signifi cance at 5% level Table 6 reports the estimates for children from rich and poor households. The rich kids’ sample consists of children whose household asset index belongs in the top 35% of the wealth distribution in the sample and vice versa for the poor kids (bottom 35% of the wealth distribution). These wealth thresholds were necessitated by the need to have suffi cient data for analysis while leaving out a sizable median segment. The table shows a positive impact of the CRI program on the achievement of both rich and poor children with a relatively higher impact on richer children. Given the point estimates, there is a difference of about 1.67 units in CRI’s impact on the overall IRT-adjusted score for rich and poor children, which indicates considerable heterogeneity of the program’s effect across students. In addition, the ATT estimates in panel B in table 6 are not statistically Page | 17 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) signifi cant owing perhaps to the relative imprecision from using a small estimation sample. The same difference in statistical signifi cance can be seen between panels A and B of Table 5. 4.5. Board Results of Grade V, 2006 - 07 Both CRI and Non-CRI school children appeared in the external board exams for Grade V on 2007. Based on their aggregate marks in these exams, we test whether there was any signifi cant difference in performance between the CRI and the Non-CRI school children. A preliminary look at the data suggests that the students in CRI partner classrooms did well on the board exams relative to the Non-CRI students. Graph 4 plots the distribution of the exam marks without controlling for child characteristics. The scores of the CRI children are distributed to the right indicating their overall better performance in the exams. A t-test for the difference in means also indicates that the students in CRI partner classrooms performed better. GRAPH 4: DISTRIBUTION OF BOARD EXAMS WITHOUT CONTROLLING FOR CHILD CHARACTERISTICS TABLE 7: DESCRIPTIVE STATISTICS: BOARD RESULTS Observations Mean Std, Error 95% Confi dence Interval Control 623 293.92 3.20 287.63 300.22 Treatment 486 321.82 4.15 313.68 329.97 Diff erence 1109 27.90* 5.15 17.78 38.01 * indicates signifi cance at 5% level (based on a t-test for diff erence in means) These preliminary fi ndings, however, do not control for individual differences in background or ability. So we compare matched students, selected through PSM. Table 8 presents the results based on different matching Page | 18 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) estimators that control for individual heterogeneity.13 The estimates are positive but not signifi cant. Hence, we fi nd no signifi cant evidence that the students in CRI partner classrooms performed any differently relative to the Non-CRI students on the board exams. TABLE 8: ATT FOR BOARD RESULTS Treatment Eff ect (atts) Standard Error+ 95% Confi d- No. of Observations ence Interval++ Treatment Control Total Stratifi cation Method 6.3 7.38 -7.1 24.3 622 808 1430 Nearest Neighbor Matching 6.7 8.26 -3.4 26.5 623 195 818 Kernel Matching Method 6.3 5.31 -4.4 17.6 623 884 1507 + Bootstrapped Standard Errors ++ Bias Corrected Confi dence Intervals The apparent contradiction between the results from the LEAPS testing instrument and the board exams can be resolved by carefully distinguishing between the two testing situations. LEAPS instrument measures the ability of a student at a particular point in time and were conducted within a couple of days’ advance notifi cation to the schools. Therefore, it gauges students on their ability in English, Urdu and Mathematics rather than their preparedness for the exam. Board exams, on the other hand, refl ect cumulative preparation from the prescribed course content and text books. In addition, questions appearing on these external exams are often quite similar to those given in the textbook exercises, commonly known as guess questions. Many students prepare for the board exams by cramming solutions to these guess questions. Hence, their marks may not be truly refl ective of their grasp of the underlying concepts but a mere refl ection of their intense preparation for the test. Hence, given the belief that cramming course content is encouraged in teacher-centered learning environments, it should not come as a surprise that Non-CRI children perform as well as the CRI children on board exams, even while the students in CRI partner classrooms perform much better on the LEAPS testing instrument. More importantly, one should note that difference between the two groups in Table 8 is insignifi cant implying thereby that, on the board exams Non-CRI children are neither better nor worse than CRI children. Thus, we can state that a student-centered class room environment does not adversely affect the performance of students on board examinations. So, parents who prefer teacher-led schools due to their supposed superiority on board results, need to consider the fact that while child-centered classrooms are shown to facilitate learning, students in such schools perform equally well on the board exams. 4.6. Sensitivity Analysis In order to check the robustness of the above results, we performed the matching exercise with alternative estimators to compute the program’s impact. The alternative estimators that we have used were the following: nearest neighbor matching (for each CRI student we picked a matching non-CRI student and measured the difference in their outcomes), radius matching and kernel-based matching. The results are similar to what was estimated before i.e. CRI impacts positively on student achievement and there is no signifi cant difference in 13 Three different methods were utilized to match students: Stratifi cation Method, Nearest Neighbour Matching and Kernel Matching Method. Page | 19 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) performance between the CRI and Non-CRI students on Board Exams. We have, in addition to our existing control variables, also tested for the presence of certain unobservables (those variables that cannot be explicitly measured) by computing an upper bound for their infl uence on the effect of CRI intervention to become meaningful. Specifi cally, one would be concerned about the validity of our results if there are certain attributes which would infl uence both the participation decision and the outcome of interest. It could be that more motivated parents send their children to CRI partner schools which in turn refl ect in higher achievement of these children, independent of the CRI intervention. Since variables like parental motivation cannot be captured or measured easily, we would have reason to suspect the fi ndings of our evaluation if it exhibited a strong presence and infl uence. In order to test for the presence of these unobservables we carry out certain sensitivity checks of the results.14 They indicate that the chances of unobservables affecting our results are reasonably small. 14 These sensitivity results are highlight in the CRI Academic Paper in the Appendices of this report. Page | 20 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 5. CONCLUSION The CRI program has been working for the last fi ve years to improve the learning environment in Islamabad government schools by making classrooms child centered. The Program has adopted a multi-pronged strategy where it trains the teachers in interactive teaching styles besides provide additional teaching aids and instructional material for classrooms. We have found that there is a signifi cant positive learning impact of the CRI program on all the curricular components of elementary education which may be explained by its effect on the classroom environment and increased student attendance in CRI partner schools. Grade IV children in CRI partner schools performed better, on average, than comparable children in non-CRI schools. This higher performance was seen in all three subject areas: English, Urdu and Mathematics, when comparing children who have had a greater exposure to the program. Based on our estimates and the empirical distribution of test scores in the sample, exposure to the CRI program improved the average student’s ranking by 4-11 percentiles above his current standing vis-à-vis other students in the cohort. To put this in context, the CRI effect translates to an improvement of ½ - 1 letter grade depending on the students’ initial position. For the median (50th percentile) student, a 5.55 points increase would translate to a 10 percentile or one letter grade improvement from D to a C (given the scheme you devised A: 90-99th percentile, B: 70-89, C: 60-69, D: 50-59). For a student at the 75th percentile, the same increase of 5.55 points would amount to an improvement of 8 percentiles in the rank and so on. The jump in student ranking varies between 5 (for those among the top 10% of students) and 10 percentiles (at the median). Another way to put the magnitude of CRI effect in context is to compare it with other schooling interventions. The Tennessee STAR experiment, for example, for which class size was reduced by 7 to 8 children (from 22 to about 15), improved test scores by about 0.21 standard deviations over a period of four years (Krueger and Diane Whitmore, 2001) compared to our estimate of the fi ve-year cumulative effect of the CRI program equaling 0.25 st. devs. The STAR program was found to have a larger impact on the achievement of black students, improved by 7-10 percentiles, than white students (3-4 percentile points) although the authors think these are likely underestimates of the true effect due to an incomplete take-up of the program. Similarly, the private school voucher program in USA evaluated in an experimental study by Howell, Wolf, Peterson, Campbell (2000) led to a 6 percentile point improvement in student’s performance from a 2-year program exposure (after adjusting for incomplete take-up). Disaggregation results by gender and wealth indicates a slight advantage in favor of boys and children of higher income households. We show that this causal relationship between the CRI program and greater learning achievements continue to exist even in the presence of unmeasured confounding variables. While we find that CRI program induces greater learning achievement in students, we also fi nd that CRI students perform no differently vis-à-vis the Non-CRI students on board exams where Non-CRI students might perform equally well due to, among other things, rote learning that commonly takes place in a teacher￾led environment. Overall, our results present a contrasting picture to previous analysis that examines non-didactic learning approaches and child learning outcomes in the developing country context. We however qualify our results, in the Pakistani perspective, where the schooling process is widely held to be associated with tedium and apathy. Page | 21 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) In such a setting we fi nd that interventions that attempt to change the very nature of these public school classrooms, rendering them interactive, child friendly and fun can indeed provide one solution for Pakistan’s education problems. A deeper understanding of the particular channels through which the CRI program is found to be effective is needed. Exploring the various links of these specifi c inputs at the elementary schooling level forms our research agenda for the future. In particular, are interactive programs such as these, able to develop child non-cognitive ability which in turn may fi nd manifestations in better cognitive achievement of children? Page | 22 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) REFERENCES ANDRABI, T., DAS, J., AND KHAWJA, A. (2002) “Test Feasibility Survey Pakistan: Education Sector”. The World Bank. Washington D.C. mimeo. CALIENDO,M., AND KOPEINIG, S. (2005) “Some Practical Guidance for the Implementation of Propensity Score Matching”, Discussion Paper, No. 1588, Bonn Germany. FILMER D. AND PRITCHETT L.(1997) “What Production Functions Show: A Positive Theory of Education Spending”, A World Bank Policy Research Working Paper, 1975. Washington D.C. NASEER, F., PATNAM, M., RAZA, R. (2007) “Are Child Centered Classrooms Effective? Impact of the CRI program on Student Learning Outcomes in Pakistan”, Lahore University of Management, Lahore. mimeo. ROSENBAUM, P. (2002) Observational Studies, New York, Springer. ROSENBAUM, P., AND RUBIN, D. (1983) “The Central Role of the Propensity Score in Observational Studies for Causal Effects”, Biometrika, 10:1. SIANESI, B. (2001) “Implementing Propensity Score Matching Estimates with STATA”, Prepared for UK STATA User Group, UK Meeting, London. WATSON, I. (2005) “The Earnings of Causal Employers: The Problem of Unobservables”, presented at the 2005 HILDA Survey Research Conference, University of Melbourne. Page | 23 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) APPENDIX Page | 24 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) TEACHER QUESTIONNAIRE (For Grade IV) Impact Evaluation of CRI Program in Islamabad Evaluation Research Study Conducted by Lahore University of Management Sciences (LUMS) in collaboration with Research Consultants (RCons) Tick here when completed ________________________ Page | 25 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section 0 Sr. No. Questions Answers Codes Instruction 1 Name and Code of enumerator Name _________________________ Code Please write down your own name and code over here 2 Name and code of School Name _________________________ Code Use school code list 3 Name and Code of Respondent Name _________________________ Code Please fi ll in the code of the respondent from the teacher roster 4 a. Have you received the CRI training? (If no skip to the next question) 1= Yes 2= No b. For How many Classes/Grades , , , , 0 = No training 1 = Grade 1 2 = Grade II 3 = Grade III 4 = Grade IV 5 = Grade V 5 Have you received the FDE’s Early Childhood Education (ECE) training? 1= Yes 2= No 6 Interview Date / / Date Month Year Write down date of interview here Page | 26 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section I: Professional Information for Teacher Sr. No. Question Answer Codes Instructions 1 How long have you been a teacher at this school? Years Months 2 How many years have you been teaching in total? If less then 1 year then write 01 Write down completed years 3 What kind of transportation do you use to get to school? ________________________ 1=Walking 2=Cycle 3=Bus 4=Motorcycle/moped 5=Bullock Cart 6=Other If more than one kind is used then write down which is more commonly used If others then specify 4 How long does it take for you to get to the school from home? 1=Less then 15 minutes 2=15 minutes to 30 minutes 3= 31 minutes to 1 hour 4=More then one hour This question is related to Q7. If teacher is travelling through cycle then how long does it take to reach school by cycle 5 Do you like teaching 1=Yes 2=No 6 Can you name these 3 [randomly picked] children in your class Note the # of children correctly identifi ed by the teacher Choose 3 children in class and ask the teacher to name them 7 Teacher’s gender. 1=Male 2=Female Page | 27 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section II: School Information for Teacher Sr. No Question Answer Codes Instructions 1 What class (es) do you currently teach? (only primary) , , , , , Record each class taught separately in the boxes If prep class code it 0. NOTE: Record Primary Classes only 2 What language do you teach math/ science Science ______________ Math ______________ 1=Urdu 2=English 3 =Urdu and English 4 =Urdu and Punjabi 5=Urdu and Seraiki 6=This subject is not taught 7=Urdu, Punjabi and saraiki 8=others ………… If others then specify 3 How important is your role in … Decisions concerning your teaching methods. 1=Very important 2=Important 3=Not important 4=Totally unimportant Write down the respondent’s role in each of the 2 options. Do not leave any box blank 4 … Decisions concerning your teaching material requirement 5 When was the last time you or your class were visited by a school inspector? (Did some government or NGO offi cial visited the school) 1=0-1 months ago 2=2-3 months ago 3=4-6 months ago 4=7-12 months ago 5 = >1 year ago 6 = never 6 When was your last contact with the education department (did you visit the education dept offi ce) 1=0-1 months ago 2=2-3 months ago 3=4-6 months ago 4=7-12 months ago 5 = >1 year ago 6 = never Page | 28 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section III: In Service Teacher Training History Sr. No. 1 23456 PERIOD Year in which training was received (Start with the most recent training) Days of training received Training provided by 1=Government/FDE’s ECE Training 2=School (school teachers) 3=Private Institute 4=CRI 5=NGO/Trust 6=Other Special topics in training (First) 1=English 2=Math 3=Science 4=Teaching methods (General) 5=Other Special Topics in Training (Second) 1=English 2=Math 3=Science 4=Teaching methods (General) 5=Other Instructions 00=No Training (Next Section) Days of training (Exclude the professional degrees like B. Ed, PTC etc) If other, then specify If other, then specify If more then one important topic were the focus of the training then mention second topic here. If other, then specify A Training Episode 1 _________ _________ _________ B Training Episode 2 _________ _________ _________ C Training Episode 3 _________ _________ _________ D Training Episode 4 _________ _________ _________ Note: In column 5 & 6 write 2 important components of the training Page | 29 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section IV: Current Household Information for Teacher Sr. No Question Answer Codes Instructions 1 How many children do you have living with you? (write down the number of children less than 15 years old) If 00 then skip to next section Please count all the children who are living with the teacher not necessarily teacher’s own children Record 0 for no children (->Section V) 2 Of these children how many are eligible to attend this school. Number of children A child is `eligible’ to attend the school if she is of the correct gender and age￾group for the school considered. 3 What schools are they attend￾ing? Child Number Gender 1= Male 2= Female School Type Codes for School Type 1=This School 2=Other CRI school in the same sector 3=Non-CRI School in the same sector 4=Other CRI School outside the sector 5=Non-CRI School outside the sector 6=Madrassa in the same sector 7=Madrassa outside the sector 8=NGO/Private in the same sector 9=NGO/Private outside the sector 10=child does not attend any school (Write only for those children which are mentioned in question 2 above.) 1 2 3 4 5 6 7 8 Page | 30 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section V: Remuneration and Contracts Sr. No. Question Answers Codes Instructions 1 Can you receive any additional amounts above the salary, such as bonuses and prizes? 1=Yes 2=No (-->5) If 2 then skip to Q5 2 What can you receive bonuses for? , Enter a maximum of two codes 1=Regular Attendance 2=Children’s performance in examination 3=Parental Praise 4=Extra Responsibilities in school 5=Other If other then specify 3 Have you ever received a bonus/prize? 1=Yes 2=No (-->5) If 2 then skip to Q5 4 When was the last time you received a bonus/ prize. 1 = 0-1 month ago 2 = 2 to 6 months ago 3 = 7 to 12 months ago 4 = More then 1 year ago 5 Besides your own salary and allowances as a teacher in this school do you have any other sources of income? From agriculture From wage employment From teaching outside hours From business From any other sources 1=Yes 2=No Please fi ll in all the boxes with 1 or 2. do not leave any box empty 6 What percentage of your total earning comes from your remuneration as a teacher in this school? 1=Less then 10% 2=10% to 25% 3=26% to 50% 4=51% to 75% 5=76% to 100% 6 = maximum 100% (Teacher salary x 100)/ teachers total income Combine teachers salary and income from other resources Page | 31 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section VI: Teacher Absenteeism Sr. No. Question Answer Code Instructions 1 Were you obliged to take any time off during the {last month} as a result of an emergency? 1=Yes 2=No (-->3) If 2 then skip to Q3 2 What kind of emergency was it? Episode Days Reason 1= Own Illness 2=Illness of Others 3=School Emergency 4=Funeral 5=Other (specify) Days means the number of days teacher was absent 1 ------- 2 -------- 3 Were you obliged to take time off during the {last month} as a result of any offi cial work (such as workshops)? 1=Yes 2=No (-->5) If 2 then skip to Q5 4 What kind of work was required? Episode Days Reason 1=Meeting 2=Workshop 3=File Request/ Complaint 4=Obtain school/offi ce provisions 5=Collecting Salary 6=Other Do not include offi cial holidays like winter break 1 2 5 Did you take any other working days off during the last month due to any other reason? 1=Yes 2=No (-->Next Section ) 6 What was the reason? Episode Days Reason 1= Personal Work 2=Household Work 3=Lack of Transportation 4=Poor Weather 5=Other Do not include offi cial holidays like winter break 1 2 Page | 32 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section VII: Teacher Time-Roster Please trace out your regular activities during the school day (but not restricted to school hours) Start from the time that the teacher wakes up in the morning to the time that he/she goes to sleep. Instruction Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Kindly note down activities of the teacher for his entire day. Start from th time when teacher wakes up until he sleeps. Time -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ Activity If more codes are needed then start from code 18 and list the activity in code list Note: Use 24 hour clock to note time Slot 9 Slot 10 Slot 11 Slot 12 Slot 13 Slot 14 Slot 15 Slot 16 Time -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ -------to------ Activity Codes for Activity 1=Class time: English 5=Class time: Science 9= Marking Homework / Tests 13=Community Activities 17=preparation for school 21=Other Code (Specify ________________ ) 2 = Class Time: Mathematics 6=Class time: Social studies 10=Assembly 14=Housework 18= Other Code (Specify ________________ ) 22=Other Code (Specify ________________ ) 3 = Class Time : Urdu 7 = Private Tuition 11=mid break/ break/free period 15= Religious Activities 19=Other Code (Specify ________________ ) 23=Other Code (Specify ________________ ) 4 = Class Time : Islaamiyat 8= Preparing for Class 12=Leisure activities outside school 16=travelling to and from school 20=Other Code (Specify ________________ ) 24=Other Code (Specify ________________ ) Page | 33 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section VIII: Classroom Facilities Please tell us the number of each of these items provided by the school for your Class IV children. Sr. no. Item Number of items Instructions 1 Math Textbooks Only note down total number of textbooks provided by school 2 English Textbooks 3 Science Textbooks 4 Urdu Textbooks 5 Religious studies Textbooks 6 Desks 7 Chairs 8 Are the following items present in the classrooms: a. Chatai/Rug 1= Yes 2 = No b. Active Learning materials c. Child art work on walls d. Blackboard Page | 34 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) School Code Use school code list Grade 1=Grade 4 2=Grade 5 Child Code Use child roster CHILD QUESTIONNAIRE (For Grade IV and V) Impact Evaluation of CRI Program in Islamabad Evaluation Research Study Conducted by Lahore University of Management Sciences (LUMS) In collaboration with Research Consultants (RCons) Tick here when completed ___________________ Page | 35 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section 0 Sr. no. Questions Answers Codes Instructions 1 Name and Code of enumerator Name ______________________ Code Please write down your own name and code over here 2 Name and code of School Name ______________________ Code Use school code list 3 Child Name Name ______________________ Code Write child name and code from child roster 4 Childs Father’s Name ______________________ 5 Childs Grade 1=Grade 4 2=Grade 5 6 Home Address Sector_________________________, Street_________________________, House No______________________ 7 Interview Date / / Date Month Year Write down date of interview here Page | 36 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section I: Basic Child Information Sr. No. Questions Answers Codes Instructions 1 What is your age? Years Months Write down age from the admission register (if available) 2 From what class have you been studying in this school Class Use 0 for KG 3 What kind of transportation do you use to get to school? ______________________ 1=Walking 2=Cycle 3=Public Bus 4=School Bus 4=Motorcycle/moped 5=Bullock Cart 6=Other Readout the list to the child If others then specify 4 How long does it take for you to get to the school from home? 1=Less than 15 minutes 2=15 minutes to 30 minutes 3= 31 minutes to 1 hour 4=More than one hour This question is related to Q6, if a child comes to school by bus then how long does it take to reach school by bus 5 Does your mother live in the same house as you. 1=Yes 2=No 6 Does your father live in the same house as you? 1=Yes 2=No 7 How educated is your mother. 1=No Education 2=Up to and including primary 3=Primary to Higher Secondary 4=Higher Secondary or higher Less than primary means that school attended but did not pass grade fi ve exams Page | 37 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 8 What’s your mother’s occupation ______________________ 1. Teacher Doctor2. 3. Armed Forces 4. Salaried Government Employee 5. Salaried Private Employee 6. Self Employed (own business) 7. Engineer 8. Politician Farmer9. 10. Housewife 11. Laborer 12. Child can not describe 13. other (_________) Do not tell these codes to child at all. 9 How educated is your father. 1=No Education 2=Up to and including Primary 3=Primary to Higher Secondary 4=Higher Secondary or higher Less than primary means that school attended but did not pass grade fi ve exams 10 What’s your father’s occupation ______________________ 1. Teacher Doctor2. 3. Armed Forces 4. Salaried Government Employee 5. Salaried Private Employee 6. Self Employed (own business) 7. Engineer 8. Politician Farmer9. 10. Housewife 11. Laborer 12. Child can not describe 13. other (_________) Do not tell these codes to child at all. 11 How many elder brothers do you have at home? Those who are permanently living with you. (Do Not include those staying some where else because of education or work) Page | 38 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 12 How many elder sisters do you have at home? Those who are permanently living with you. (Do Not include those staying some where else because of education or work) 13 How many days do you study in a week at home Days Ask child how many days he studies in a week at home if he says “Daily”, write 7 days. But if he says 2 or 3 days then again ask and confi rm either 2 or 3 days and write the answer. 14 How much time do you spend in studies daily : hrs minutes Write the total duration spent on studies daily like if child said some time he spent 2 hrs and some time 3 hrs then write the average duration i.e. two hours thirty minutes (02:30) 15 How do you study 1 Alone 2. with the help of family 3Tuition 1=Yes 2=No Ask each category separately 16 What is the highest grade that you wish to study Write the highest grade that child whish to study i.e. if chid say BA then write 14 an if response is Masters than write 16. 17 What do you want to be in your life. ______________________ 1. Teacher Doctor2. 3. Armed Forces 4. Salaried Government Employee 5. Salaried Private Employee 6. Self Employed (own business) 7. Engineer 8. Politician Farmer9. 10. Housewife 11. Laborer 12. Child can not describe 13. other (_________) Do not tell these codes to child at all. 18 Have you been punished in school last week 1=Yes 2=No (Skip --> 21) If 2 then skip to Q21 Page | 39 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 19 What type of punishment was given? , _______________________ 1. Physical punishment 2. Stood in corner/top of table 3. Break or game period was not allowed 4. Given Extra work 5. other (________) can enter more than one code 20 Why was this punishment given? ______________________ 1. Mischief/Fight 2. Did not study 3. Came to school late 4. Extra holidays 5. other (________) 21 Do you feel like going to school 1=Yes 2=No 22 How do you interact with your peers: a. Do you sit in groups in class b. Do you take part in activities along with your classmates 1= Yes 2 = No 23 Have you ever approached your class teacher outside of class over the last two weeks 1= Yes 2 = No 24 Who do you share your problems with Teacher Siblings Classmates Parents Other (specify) (________) 1= Yes 2 = No 25a In the last two weeks did your teacher show appreciation for your work 1= Yes 2 = No (Skip to 26) Page | 40 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) 25b How Stars marking/good Candy Praise in class Clapping Other (___________) 1= Yes 2= No 25c How many times 26 Does your teacher use other materials to teach apart from textbooks Posters Blocks Plants Other 1= Yes 2= No 27 Do you enjoy the following in school 1. Assembly 2. Study 3. Break 4. Game Period 0. Not applicable 1. Enjoy a lot Enjoy2. 3. Enjoy a little 4. Do not enjoy at all Ask for each component separately. Two or more component could be coded with one code. Page | 41 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) Section II: Assets (The assets that are own by you) Sr. No. Asset Category Does your household have ______________________ Sr. No. Asset Category Does your household have ______________________ 1=Yes 2=No Number 1=Yes 2=No Number 1 Beds 11 Tractor 2 Charpai 12 Cattle (horse, buff alo, cow) 3 Air Conditioner 13 Goats/sheep 4 Television 14 Motorcycle/Scooter 5 Refrigerator 15 Car/Taxi/van/pickup 6 Bicycle 16 Telephone 7 Plough 17 Tube well 8 Radio/Tape Recorder 18 Mobile Phone 9 Washing Machine 19 Fans 10 Chairs 20 Computer Page | 42 IMPACT EVALUATION OF CRI - PAKISTAN ( 2002-2007) SECTION IV: Anthropometrics Sr. No. Questions Answers Instructions 1 Child Weight pounds Write down child weight in words here ______________________ With shoes but with out pullovers Write down child weight in numeral as well as in words 2 Child Height centimetres Write down child height in words here ______________________ Without shoes Write down child height in numeral as well as in words 3 Child Disease/ Morbidity: Number of times child fell sick Days of school missed due to illness/sickness How many times did child fall ill during the last three months IMPACT EVALUATION of Children’s Resources International, Pakistan (2002 - 2007) PREPARED BY: Lahore University of Management Sciences (LUMS) Children’s Resources International, Pakistan Head Office: # 66, St. 89, G-6/3, Islamabad Tel: (92 51) 2877941-2, Fax: (92 51) 2274140 Sub Office: 192-P, Block 2, P.E.C.H.S, Karachi Tel: (92 21) 4386967, Fax: (92 21) 4386967 info@cripk.org OR www.cripk.org A REPORT ON 9 789699 029134