EVALUATION AND RESEARCH FINAL REPORT APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Prepared under Contract No.: GC-10F-0033M/AID-OAA-M-13-00010 , Tasking N 015.02 Cover photo credit: Alicia Menendez USAID READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Endline Report JUNE 2022 Prepared under Contract No.: GC-10F-0033M/AID-OAA-M-13-00010, Tasking N 015.02 Submitted to: Benjamin Silla, USAID COR Submitted by: Alicia Menendez, Gregory Haugen, Alejandro Ome, Mayumi Rezwan, Ninar Taha and Paige Pepitone Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Varuni Dayaratna DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | i TABLE OF CONTENTS EXECUTIVE SUMMARY................................................................................................................................................. iv INTRODUCTION ............................................................................................................................................................ 1 EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW.................................................................. 5 FINDINGS.........................................................................................................................................................................11 CONCLUSIONS..............................................................................................................................................................31 RECOMMENDATIONS.................................................................................................................................................33 REFERENCES....................................................................................................................................................................35 ANNEX I: EVALUATION STATEMENT OF WORK............................................................................................37 ANNEX II: EVALUATION METHODS AND LIMITATIONS .............................................................................49 ANNEX III: VIOLENCE INDICATORS .....................................................................................................................64 ANNEX IV: ADDITIONAL ANALYSIS .....................................................................................................................65 ANNEX V: BALANCE BETWEEN TREATMENT AND CONTROL GROUPS.............................................68 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST ....................................................................71 TABLE OF TABLES Table 1: Main Outcome Indicators, by Treatment Status....................................................................................... vi Table 2: Main Outcome Indicators, by Student Gender........................................................................................ vii Table 3: List of Data Collection Activities................................................................................................................... 7 Table 4. Tracked Learners Sample ..............................................................................................................................10 Table 5. Main Outcome Indicators, by Student Gender........................................................................................12 Table 6: Main Outcome Indicators, by Treatment Status......................................................................................13 Table 7: Moving to Attend Secondary School ..........................................................................................................14 Table 8: Secondary School Enrollment – School Type...........................................................................................14 Table 9: Completed Level of Education before Dropping Out of School Enrollment....................................15 Table 11: Factors associated with school dropout. Multivariate regressions...................................................16 Table 11: Reasons for dropping out of school self-reported by learners..........................................................17 Table 12: Percentage of secondary learners receiving support to pay school fees.........................................18 Table 13: Percentage of secondary learners receiving support to pay school expenses other than fees..19 Table 14: Activities among dropouts..........................................................................................................................19 Table 15: Aspirations and expectations at baseline, students and dropouts.....................................................20 Table 16: Aspirations and expectations at baseline and endline in treatment and comparison communities......................................................................................................................................................................22 Table 17: Secondary school climate according to students...................................................................................23 Table 18: Percentage of Students and Dropouts that Witnessed Violence in the Community and at Home..................................................................................................................................................................................23 Table 19: Prevalence of Emotional, Physical and Sexual Violence at Endline, Students and Dropouts......24 Table 20: Descriptive statistics by teacher cohort..................................................................................................25 Table 21: Retention rates by teacher cohort............................................................................................................26 ii | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Table 22: Sociodemographic characteristics by retention status.........................................................................27 Table 23. Treatment and Comparison Communities in Machinga District.......................................................50 Table 24. Treatment and Comparison Communities in Balaka District............................................................51 Table 25: Evaluation Matrix...........................................................................................................................................52 Table 26. Potential Progress for Students.................................................................................................................55 Table 27. Main Outcome Indicators, by Student Gender......................................................................................65 Table 28. Difference in Difference Estimation of Aspirations and Expectations – Machinga and Balaka Districts..............................................................................................................................................................................66 Table 29. Difference in Difference Estimation of Aspirations and Expectations – Balaka District..............67 Table 30. Balance between Treatment and Control...............................................................................................68 TABLE OF FIGURES Figure 1: Theory of Change............................................................................................................................................. 2 Figure 2: Average number of teachers per school ..................................................................................................25 Figure 3: Teachers’ satisfaction with building and facilities of the AMAA CDSS..............................................29 Figure 4: Teachers’ satisfaction with the school housing.......................................................................................30 Figure 5. AMAA Theory of Change Diagram: The Whole of Girl Approach...................................................40 READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | iii ACRONYMS AMAA Apatseni Mwayi Atsikana Aphunzire CDSS Community Day Secondary Schools DEC Development Experience Clearinghouse DFID Department for International Development DID Difference-in-Difference E3/GenDev Office of Gender Equality and Women’s Empowerment EDR Evaluation Design Report EMIS Education Management Information System FGD Focus Group Discussion GRS Grassroot Soccer IKI Invest in Knowledge, Inc. JICA Japan International Cooperation Agency LGL Let Girls Learn MoEST Malawi’s Ministry of Education, Science and Technology OU Operating Units SCI Save the Children International SOW Scope of Work SRGBV School-Related Gender-Based Violence QED Quasi-Experimental Design USAID U.S. Agency for International Development iv | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA EXECUTIVE SUMMARY Apatseni Mwayi Atsikana Aphunzire (AMAA) was a USAID-funded project, led by Save the Children (SC), and implemented in collaboration with Concern Worldwide (CW) Grassroot Soccer (GRS). AMAA was supported by USAID’s Let Girls Learn Challenge Fund awarded by the Gender Equality and Women’s Empowerment Hub. AMMA constructed 11 Community Day Secondary Schools (CDSS) – 5 in Machinga District and 6 in Balaka District– and provided scholarships to vulnerable boys and girls. NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, conducted an external and independent evaluation and research studies of AMAA CDSS construction Little empirical work has been done anywhere in the world on school construction, particularly for the case of secondary schools, and the studies that have been published demonstrate significant variability. While some evidence suggests the construction of schools may increase enrollment, particularly for girls, it is clear that effects will be highly dependent on context, and that additional evidence is needed, which helps to motivate the present study. This evaluation is an important opportunity to increase the evidence in general, and to advance our understanding of the Malawian case in particular. EVALUATION QUESTIONS USAID Equality and Women’s Empowerment Hub proposed the list of evaluation and research questions below which guided this evaluation. (1) What effect does embedding a new CDSS in a community have on (1) primary school completion and transition; and (2) secondary school enrollment, attendance, promotion and retention of girls and boys in the community? (2a) What effect does embedding a new CDSS in a community have on the attitudes and expectations of girls, boys, parents, and communities in the areas served by the AMAA-constructed new schools? (2b) What are the attitudes and expectations of (1) girls and boys who have dropped out of school; and (2) girls and boys who are in the new CDSSs? (3) What are the reasons for girls and boys to: (1) drop out of school; (2) repeat; and (3) be held back? (4) How are the CDSSs helping girls to overcome identified barriers, such as: education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school, and lack of space in secondary schools? Are there additional barriers to accessing schools faced by girls? (5a) What effect do the new CDSSs have on the perceptions of learners regarding physical and socio￾emotional aspects of safety? (5b) What are the perceptions of girls and boys dropping out of school regarding physical and socio￾emotional aspects of safety? (6) What experience do new CDSSs have on attracting and retaining teachers? READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | v (7) Are girls, their parents/guardians, and community members satisfied with the building facilities of the new schools? METHODOLOGY OVERVIEW The evaluation design includes different methods depending on the question to be addressed. Some questions are answered using quantitative data only, while other questions include the use of qualitative information. Quantitative and qualitative data complement each other to gain details and understanding. The methods required quantitative and qualitative data gathered from learners, parents, secondary teachers, and leaders in communities where the secondary schools are being built (Treatment) and in communities without secondary schools but are similar otherwise (Comparison). COMMUNITY SELECTION AND ALLOCATION OF AMAA CDSSS NORC took advantage of the site selection process that considered a group of similar communities as candidates for treatment. Treatment was only assigned to a subgroup of communities, allowing NORC to draw comparison groups from the rest of the list. A committee made up of USAID/Malawi, Save the Children, and MoEST officials, supported by technical teams, selected the Treatment communities to receive the AMAA secondary schools. USAID/Malawi selected the initial pool of candidate communities considering primary school pass rate, Standard 8 enrollment, located at least 10km away from another CDSS. The committee then selected communities from this list based on their own preferences. Each selected community was then assessed on technical suitability with respect to the following criteria: site large enough to cater to the size of the structures to be constructed; land voluntarily donated by the community and land users, not owned by the church; soil and land slope suitable for construction; nearby water source; and environmental impact of construction. Some selected communities were rejected on technical grounds1 and had to be replaced by others included in the original list of candidates. This occurred several times, giving the selection into treatment process a high degree of (unintended) randomness and allowing the evaluation team to use the initial list as a source for selecting comparison communities. The reader needs to bear in mind that this allocation of communities to treatment was not the result of a randomized control trial. Although at baseline, the treatment and comparison communities appear to be similar, they are not statistically identical. We used a process called entropy balancing (Hainmuller, 2012) to improve the comparability of treatment and comparison groups, so that the comparison group has a more similar distribution to students in treatment communities on observed characteristics that may influence outcomes (e.g., age, mother’s education, etc.). This helps to overcome the confounding effects of potential selection bias. 1 The technical issues that affected selection of communities to receive treatment are not likely to have incidence on or be associated with school enrollment or other outcomes of interest in this study and, therefore, do no create bias problems. vi | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA DATA COLLECTION The evaluation team used quantitative and qualitative data collected in each community included in the study (11 Treatment and 12 Comparison). NORC and its local partner Invest in Knowledge (IKI) created a panel of learners that were attending Standards 5, 6, 7, and 8 at baseline and followed yearly until the 2021 endline. Students in the tracked sample were randomly selected and chosen to include 10 male and 10 female students from each standard in each community, for a total of 80 students per community. A sample of caregivers was also randomly selected at baseline from a sub-sample of the tracked students. After baseline, annual follow-up data included: (1) administrative school data to determine the current enrollment status of all students that were in Standard 5-8 at baseline; (2) student surveys administered to students from the sample who have either dropped out or transitioned from primary school on to Form 1 since the 2018 baseline survey; (3) a survey administered to the caregivers of students in the tracked sample who have dropped out since the 2018 baseline; and (4) interviews with teachers and head teachers in newly constructed schools in the treatment communities. At endline (2021) we additionally included interviews with all students in our tracked sample, interviews with the mothers of tracked girls in our sample who had either dropped out or enrolled in secondary school, and interviews with mothers of girls in Standard 5-8. Levels of attrition were similar for treatment and comparison communities and for girls and boys. The level of status attrition was particularly low, at just 7 students out of the full sample of 1,823 (0.4 percent), while the level of interview attrition is higher and typical of a long-term panel study, with a total of 260 of 1,823 students (14.3 percent). FINDINGS School enrollment and dropouts. The newly built AMAA CDSSs have increased Malawi’s capacity to educate secondary school students, particularly in the rural areas of Machinga and Balaka, where there were no public secondary schools before. In the treatment communities, almost 70 percent of the secondary school students are enrolled in the local AMAA CDSS. The benefits of these new CDSSs have reached the comparison communities as well; more than 20 percent of the secondary school students in comparison communities are attending also AMAA CDSS. Table 1: Main Outcome Indicators, by Treatment Status Treatment Pct. Comparison Pct. Diff. Treat-Comp Student is a dropout at endline 38.9 41.1 -2.2 Student was ever a dropout 39.8 42.4 -2.5 Student is still in primary school at endline 26.1 23.2 2.8 Student is in secondary school at endline 47.0 47.7 -0.7 Ever made it to secondary school 50.5 52.8 -2.3 *** p < 0.01, ** p < 0.05, * p < 0.1. All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | vii We observed a slightly lower -2.2 percentage points- reduction in dropouts in the treatment communities; however, the difference is small and not statistically significant. The proportion of students finishing primary school or attending secondary education are also very similar between community groups. However, we do see significant differences in other important aspects. While 7.6 percent of students in treatment communities attend private schools, the percentage is 18.5 among students in comparison communities. The difference in the number of students that had to move away from home to attend secondary schools is also large and significant. While almost 29 percent of the students in treatment communities moved, the percentage of students in comparison communities was significantly higher, reaching over 46 percent. Barriers to Stay in School. The AMAA schools offer access to secondary school to rural students, however dropout rates in the communities we studied continue to be high (38.9 percent of the students in our sample dropped out of school between 2018 and 2021). Dropping out of school is more common among girls than boys. In our sample, around 35 percent of boys left school during the period under study, compared to over 42 percent of girls. Overage students are also more likely to drop out of school than others. Low socioeconomic status, measured as household assets ownership and the education of the mother, is associated with higher risk of dropping out. Table 2: Main Outcome Indicators, by Student Gender Overall Girls Boys Diff. Girls￾Outcome Percent Percent Percent Boys Student is a dropout at endline 38.7 42.3 34.9 7.5 *** Student was ever a dropout 39.7 43.3 36.0 7.3 *** *** p < 0.01, ** p < 0.05, * p < 0.1. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. Poverty is the main barrier that prevents students from continuing their education. Lack of resources to pay for school fees and other education costs is the most mentioned reason (over 60 percent) for dropping out of school. AMAA schools have contributed to reducing the problem by offering bursaries (30 percent of students in AMAA schools receive financial support to cover fees vs. 14 percent in other schools) and by reducing the need to move to a different area to attend school, which increases education costs. The second most common reason for leaving school was pregnancy. More than a quarter of the girl dropouts reported pregnancy as the reason for abandoning their education. If we also include marriage and having/nursing a baby, this increases to a third of girl dropouts. A common pipeline to dropout that stakeholders described involved girls entering sexual relationships, getting pregnant, sometimes getting married, and then dropping out of school. However, being overage in school is quite common and an important contributing factor to the pregnancies problem. Among girls who dropped out before reaching secondary school and have ever been pregnant, around 65 percent had their first pregnancy at age 18 or older. Attitudes and Expectations The attitudes and expectations of girls, boys, parents, and the community in general regarding education, work, marriage, and children were extremely optimistic at baseline. An overwhelming majority of students aspired and expected to attend university, work in their prefer job by age 25, and only get married after 28 years of age. Although aspirations and expectations were slightly higher among those students that remain enrolled in school than among dropouts, they viii | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA were high for everyone and dropping out does not seem related to lack of aspirations or low expectations, at least as measured. Comparing aspirations and expectations between baseline and endline, we observe that an even larger fraction of students at endline said they wanted to attend college/university, and there are no differences between treatment and comparison communities. However, the expectations were lower from baseline to endline, as fewer students thought they would actually be able to reach their desired education levels. The drop in expectations is more pronounced among those already in secondary school than among students still in primary school, suggesting students’ expectations adjust to reality as they age and move through the education system. Students in treatment communities do not show higher expectations than their peers in comparison communities. Violence Perceptions and Experiences. The students report higher perception of safety and more parental and community involvement in AMMA schools. The results suggest AMAA CDSSs are doing a slightly better job than other secondary schools offering a positive school climate. We asked secondary school students and dropouts about witnessing violence by adults in the community and by a caregiver -against spouse or partner or against siblings. In general, the reports indicate violence is common. A higher percentage of dropouts reported witnessing violence by adults in their community than secondary school students. However, witnessing caregiver physical violence at home is almost 8 percentage points more common among children enrolled in secondary school at endline. The prevalence of secondary school students and dropouts being victims of emotional and physical violence are very similar, around 31 and 54 percent, respectively. However, dropouts reported having been victims of sexual violence in higher proportions than those enrolled in secondary school (33.5 vs 28.3 percent). AMAA Teachers. Schools have been able to attract enough teachers to replace the ones that left. In fact, Machinga has been increasing the number of teachers per school since 2019. However, the quality of incoming teachers seems to be deteriorating to some extent, as teachers in cohort 3 have less education and a somewhat less experience than cohort 1 teachers. Teacher retention is relatively high but declining, and for teachers that stayed at least one year since 2019, the retention between 2020 and 2021 is comparable to previous studies for Malawi. In terms of teacher quality, teachers that leave school have more education than teachers that stay, which suggests that the former have better outside options than the latter. Finally, teachers that leave the AMAA schools are more likely to live far from school, which highlights the importance of housing options close to schools as a determinant of teacher retention. Satisfaction with AMAA CDSSs Facilities. Students, their caregivers, teachers and head teachers, and community leaders expressed their satisfaction with the new CDSS facilities. There is agreement that students have benefitted from the construction of new CDSSs closer to their homes. Stakeholders suggested that when it is easy for a student to physically travel to school, there is no need for additional transportation and boarding costs, which are important obstacles to secondary school enrollment and attendance. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | ix Respondents expressed appreciation for the CDSS appearance and construction. They find the AMAA schools modern, beautiful, well-ventilated, well-lit, and properly furnished. Separate latrines for teachers and students, and for males and females, provide privacy and promoted adequate hygienic conditions, especially during girls’ menstrual cycles. This is a highly appreciated feature the AMMA schools. However, stakeholders were disappointed by a lack of several amenities in the CDSS, such as electricity, fences to clearly demarcate school boundaries, a library, and instructional materials such as textbooks and lab equipment. A main concern among teachers relates to the insufficient supply of housing units, as well as housing units that are smaller than needed, and lack electricity and water. Given the lack of quality housing alternatives around these rural communities, this affects teachers’ satisfaction, and possibly retention. RECOMMENDATIONS Having enough schools close to the rural population increases access to secondary education and reduces inequalities. School construction efforts should be continued in other areas. Having schools is a necessary condition to increase education, but it is not sufficient. Poverty limits the ability of families to keep their children in school. Although in 1994 Malawi introduced Free Primary Education, schools collect some fees from primary students; secondary schools, such as AMAA CDSSs, charge around MKW 10,000 per term; and, in addition, families need to cover for other costs, such as uniforms or exams fees. AMAA provided bursaries to some secondary students which helped but more support is needed to keep girls and boys enrolled in school. In addition to poverty, girls face an additional challenge to stay in school: pregnancies. Policies that promote age-appropriate sexual and reproductive health education for girls and boys should be a priority. This would not only affect school enrollment, but also reduce the transmission of HIV and other STIs. At the same time, stakeholders should recognize that the problem of pregnancies is tied to the high percentage of students who are overage. Policies should aim to ensure children start school on time and reduce grade repetition rates, thereby reducing the high prevalence of students who are well beyond primary school age but still in Standards 5-8. For example, 22 percent of students in Standard 8 at baseline were age 18 or older, when the expected age for Standard 8 is 12. This means that policies aimed at improving secondary school transition rates would be remiss if they do not begin by addressing the issues with primary schools that are causing most students to be well above the expected age for their grade. Finally, attention is needed to improve teachers’ housing options. Finding teachers for the new schools is the responsibility of the MoEST and is not easy. Many teachers do not want to live in rural areas far from the trade centers, health facilities, and recreational facilities that are available in urban areas. The construction of only two teacher houses at each school, and the limited amenities they provide, appears insufficient to attract and retain quality teachers. Increasing the number and size of the houses and adding features to increase comfort, could be a strategy for attract qualified teachers and reducing turnover. Alternatively, monetary, and non-monetary incentives could be explored, though the evidence suggests housing near the school is essential. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 1 INTRODUCTION PROJECT BACKGROUND The Malawi education system involves an 8-4-4 structure: 8 years of primary school, 4 years of secondary school, and 4 years of tertiary school. While the country has seen a growth in primary school enrollment in recent decades, enrollment rates in secondary school remain low. School fees and other education costs, long distances to school, early pregnancies, lack of schools and trained teachers, and family responsibilities are all factors that can contribute to a student dropping out. Lack of space in existing public facilities limits enrollment, as well. Increasing enrollment has been a key focus of national policy in recent decades. The transition to secondary school is a particularly difficult challenge. In part, transition rates from primary to secondary education are low because there are not enough secondary schools. As primary school completion continues to increase, demand from qualified students for secondary school continues to rise despite limited spots (Chimombo, 2014). In 2016, 193,795 out of 255,583 (75.82%) students passed the Primary School Leaving Certificate Examination (PSLCE) (Nyasa Times, 2017), which determines eligibility for secondary school. However, there were spots for just over half of these students in public schools (McConnell, 2016). Moreover, infrastructure and supplies such as classrooms, textbooks, and toilets remain inadequate. Only 21% of children of secondary school age complete secondary school (EMIS, 2015-2016). Apatseni Mwayi Atsikana Aphunzire (AMAA) was a USAID-funded project, led by Save the Children (SC), and implemented in collaboration with Concern Worldwide (CW) Grassroot Soccer (GRS). AMAA was supported by USAID’s Let Girls Learn Challenge Fund awarded by the Gender Equality and Women’s Empowerment Hub2. NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, conducted an external and independent evaluation and research studies of AMAA secondary school construction. AMAA was implemented in five districts across Malawi: Mzimba, Balaka, Machinga, Phalombe, and Chikwawa. The project targeted over 60,000 girls in both upper primary and secondary schools. AMAA had four main goals: (1) to improve the ability of girls to affect change in self and others; (2) to enhance the creation of an enabling environment for girls’ adolescent development and education achievement; (3) to strengthen the quality of teaching; and (4) improve access to secondary schools. Due to USAID and Malawi’s Ministry of Education (MoEST) priorities, as well as the range of girls’ education activities already existing at the local level, AMAA delivered district-specific activities in Malawi to mitigate barriers to girls’ enrollment and retention in school. In the districts of Balaka and Machinga, the project constructed 11 Community Day Secondary Schools (CDSS) – 5 in Machinga and 6 in Balaka – and provided scholarships to vulnerable boys and girls. NORC’s evaluation and research of the AMAA program is limited to the secondary school construction in these two districts and will focus on both girls and boys. 2 Formerly, the Office of Gender Equality and Women’s Empowerment 2 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA The AMAA CDSS were built during 2018 and 2019 and their inauguration dates are as follows: Machinga District Mwalasi CDSS September 10, 2018 Ntalala CDSS September 10, 2018 Kayuni CDSS September 10, 2018 Nambaiko CDSS September 17, 2018 Matanda CDSS January 28, 2019 Balaka District Makanjira (Mapu) CDSS October 14, 2019 Namikombe CDSS September 14, 2019 Chilanga CDSS September 16, 2019 Njerenje CDSS September 16, 2019 Namichere CDSS September 30, 2019 Mzimundilinde CDSS January 3, 2020 The evaluation questions and design of this study are informed by the Theory of Change model presented in Figure 1, created by NORC based on conversations with and documents received from Save the Children International and USAID. Figure 1 maps the causal links between AMAA’s construction of new Community Day Secondary Schools and impacts of interest. The model illustrates the processes by which school construction leads to immediate changes in school capacity and proximity, which in turn leads to beneficial outcomes for girls and boys in the surrounding areas. Figure 1: Theory of Change INPUTS 11 Community Day Secondary Schools constructed in Balaka and Machinga districts OUTCOMES Decrease the physical and emotional safety hazards related to traveling to school Increase time for students to attend class, complete homework, interact with teachers, and participate in extracurricular activities Increase interaction of teachers with parents and community members Increase student, parent, and community interest in and expectation of girls (and boys) to continue their education IMPACT Increase secondary school enrollment, attendance, promotion, and retention for girls and boys Increase primary school retention and completion OUTPUTS Increase the number of students who can attend secondary school Decrease the distance and travel time for students to attend secondary school Schools embedded in the communities they serve READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 3 Outputs. Building new Community Day Secondary Schools in Balaka and Machinga districts increases the capacity of the region to educate secondary school students. Additionally, introducing a secondary school into a community where there was not one previously reduces the distance that students need to travel from their homes to access secondary education. It also ensures the school they attend is located within their community and is, therefore, more accessible to their caregivers and other members of their community. Outcomes. These outputs are expected to have immediate effects on students and their communities. A secondary school closer to home reduces travel time and the costs associated with attending school. Students who may have needed to board near a school before can now live at home, making secondary education less costly for the family. A shorter commute frees up time for students—that time may be used to study, interact with school staff and other students, participate in extracurricular activities, or help the family at home. Furthermore, a shorter commute to school reduces the risks of physical and emotional violence that students, particularly girls, may encounter in the way to school. Closer proximity to a secondary school may also affect the wider community. Reducing the distance between school and home brings teachers and school staff closer to the families and communities of their students. Secondly, a new school can place education in the forefront of community members’ minds—raising interest and expectations for the future of their children’s education and employment opportunities. Impacts. Having identified likely immediate changes to the realities of students seeking secondary education, we can envision the final effects. First, we expect to see more students attending secondary school. More students can afford to attend a day school versus a boarding school, and acceptance rates into secondary schools should increase as school capacity grows. Second, a reduction in costs, safety risks, and travel time to school can translate into better attendance and retention rates. Third, changes in the frequency of interactions between the school and the community could strengthen the web of support needed for a student to stay and succeed in school. Given these impacts at the secondary school level, we can surmise there could also be impacts on younger students. If previously primary students failed to finish school because they did not see potential for them to attend secondary school, their motivation to complete primary school may improve and primary school retention and completion rates may also rise. We would be remiss to not mention that improved educational outcomes can reverberate into many other aspects of students’, especially girls’, lives. Continuing education can delay marriage, delay pregnancy, and improve employment opportunities for a girl, and ultimately improve the lives of her children, including their own educational achievement. However, these long-term impacts are beyond the scope and time period of this study. EVIDENCE ABOUT SCHOOL CONSTRUCTION The existing literature on secondary school construction is sparse. A study of a primary school construction program in Indonesia found that each primary school built per 1,000 children led to a modest average increase of 0.12 to 0.19 years of education and 1.5 to 2.7 percent in wages (Duflo, 2001). Another study of the same program ascertained significant but mixed macroeconomic effects. The program was estimated to generate a 10 percent greater proportion of primary school graduates in 4 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA the work force and a 6.6-7.5 percent increase in labor force participation. However, it also produced a 4-7 percent decline in wages (Duflo, 2004). Another study found a statistically significant but modest relationship between distance to school and enrollment in 13 of 21 countries measured for primary school and 7 for secondary school. However, there was large variability among different countries, suggesting the importance of context in how much distance to school impacts enrollment. Cameroon, Chad, and Benin, which are all Western and Central African countries with particularly low school enrollment, showed the highest correlation. (Filmer, 2007). Other studies showed varying results in whether distance is significantly correlated with enrollment (Handa, 2001; Burke, 2004; Lavy, 1996; Younger, 2000). For example, Handa finds that building more schools in rural Mozambique is correlated with higher primary school enrollment, though it may be less cost-effective than adult literacy campaigns for achieving this goal. On the other hand, Younger finds that travel time to school is not a statistically significant predictor of primary school enrollment status in rural Peru, though possess insufficient data to test this for secondary school enrollment, and finds measures of local school quality are better predictors of secondary enrollment than costs to attend. However, these papers only measured the correlation between distance and enrollment and do not determine the causal effects of school construction, or account for external factors that could impact this correlation. A randomized control trial in Afghanistan on the implementation of “girl-friendly” primary schools, which controls for all other factors, found significant effects from primary school construction. Placing a school in a village led to a 52 percentage point increase in enrollment for girls from the baseline level of 18 percent, and a 35 percentage point increase for boys from the baseline level of 35 percent, eliminating the gender gap in enrollment (Burde, 2013). Another “girl-friendly” primary school construction program in Burkina Faso augmented enrollment rates by 18.5 percentage points and a 4.7 greater percentage point increase for girls, while also improving test scores by 2.2 standard deviations. (Kazianga, 2013). These last two studies indicate that making schools more amenable to girls increases enrollment, particularly because the Burkina Faso study found that a large portion of the schools’ effects were due to their “girl-friendly” characteristics. However, having a school closer to home does not automatically warrant access and enrollment. More schools are necessary to increase enrollment, but they might not be sufficient. Other barriers, such as school fees and early marriage (Baird et al., 2011; Omoeva et al., 2014; Birchall, 2018; among many), are important as well. Keeping girls in school in the developing world has been an area of particular interest, and many different initiatives have taken place in the past and continue today. These include conditional and unconditional cash transfers (for Malawi, see Baird et al., 2011), programs that focus on addressing girls’ health, such as a study that analyzed the impact of deworming on school attendance (Miguel and Kremer, 2004), and on mitigating child marriage, a significant cause of girls dropping out (Wodon et al., 2017). Other approaches have engaged traditional authorities and societal institutions to change cultural norms surrounding female education (Keleher, 2008). In sum, little empirical work has been done anywhere in the world on school construction, particularly for the case of secondary schools, and the studies that have been published demonstrate significant variability. While some evidence suggests the construction of schools may increase enrollment, particularly for girls, it is clear that effects will be highly dependent on context, and that additional evidence is needed, which helps to motivate the present study. This evaluation is an important READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 5 opportunity to increase the evidence in general, and to advance our understanding of the Malawian case in particular. EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW The AMAA team, USAID/Malawi, USAID Bureau for Africa, the USAID Gender Equality and Women’s Empowerment Hub, the Education Center and the Environment, Energy and Infrastructure Center held conversations about the objectives of the study and possible research questions. The NORC evaluation team participated in some of those discussions. Based on those conversations, USAID Equality and Women’s Empowerment Hub proposed the list of evaluation and research questions below. EVALUATION QUESTIONS Question 1: What effect does embedding a new CDSS in a community have on (1) primary school completion and transition; and (2) secondary school enrollment, attendance, promotion and retention of girls and boys in the community? Question 2: (2a) What effect does embedding a new CDSS in a community have on the attitudes and expectations of girls, boys, parents, and communities in the areas served by the AMAA-constructed new schools regarding: Primary school retention and completion and transition to secondary schools; Interest in continuing girls’ and boys’ education and/or vocational training; Marriage and pregnancy; and Future work/employment/career? (2b) What are the attitudes and expectations of (1) girls and boys who have dropped out of school; and (2) girls and boys who are in the new CDSSs regarding: Perceived value of attending school and education as experienced thus-far; Future education; Marriage and pregnancy; and Future work/employment/career? Question 3: What are the reasons for girls and boys to: (1) drop out of school; (2) repeat; and (3) be held back? Question 4: How are the CDSSs helping girls to overcome identified barriers, such as: education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school, and lack of space in secondary schools? Are there additional barriers to accessing schools faced by girls? Question 5: (5a) What effect do the new CDSSs have on the perceptions of learners regarding physical and socio￾emotional aspects of safety? 6 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA (5b) What are the perceptions of girls and boys dropping out of school regarding physical and socio￾emotional aspects of safety? Question 6: What experience do new CDSSs have on attracting and retaining teachers? Question 7: Are girls, their parents/guardians, and community members satisfied with the building facilities of the new schools? These questions guide the evaluation presented in this report. METHODOLOGY OVERVIEW The evaluation design includes different methods depending on the question to be addressed. Some questions are answered using quantitative data only, while other questions include the use of qualitative information. Quantitative and qualitative data complement each other to gain details and understanding. The approach and technical details that the evaluation team used to answer each of the research and evaluation questions is included in Annex II, and briefly described in when presenting the findings. The methods required quantitative and qualitative data gathered from learners, parents, secondary teachers, and leaders in communities where the secondary schools are being built (Treatment) and in communities without secondary schools but are similar otherwise (Comparison). We describe below details about community selection, samples, and data collection. COMMUNITY SELECTION AND ALLOCATION OF AMAA CDSSS NORC took advantage of the site selection process that considered a group of similar communities as candidates for treatment. Treatment was only assigned to a subgroup of communities, allowing NORC to draw comparison groups from the rest of the list. The selection process for treatment communities is detailed below, followed by NORC’s process for selecting comparison communities. A committee made up of USAID/Malawi, Save the Children, and MoEST officials, supported by technical teams, selected the Treatment communities to receive the AMAA secondary schools. USAID/Malawi selected the initial pool of candidate communities using the following criteria: primary school pass rate, Standard 8 enrollment, located at least 10km away from another Community Day Secondary School (CDSS). The committee then selected communities from this list based on their own preferences. Each selected community was then assessed on technical suitability with respect to the following criteria: site large enough to cater to the size of the structures to be constructed; land voluntarily donated by the community and land users, not owned by the church; soil and land slope suitable for construction; nearby water source; and environmental impact of construction. Some selected communities were rejected on technical grounds3 and had to be replaced by others included in the original list of candidates. This occurred several times, giving the selection into treatment process a high degree of 3 The technical issues that affected selection of communities to receive treatment are not likely to have incidence on or be associated with school enrollment or other outcomes of interest in this study and, therefore, do no create bias problems. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 7 (unintended) randomness and allowing the evaluation team to use the initial list as a source for selecting comparison communities. The reader needs to bear in mind that this allocation of communities to treatment was not the result of a randomized control trial. Although at baseline, treatment and comparison communities appear to be similar, they are not statistically identical. We used a process called entropy balancing (Hainmuller, 2012) to improve the comparability of treatment and comparison groups, so that the comparison group has a more similar distribution to students in treatment communities on observed characteristics that may influence outcomes (e.g., age, mother’s education, etc.). This helps to overcome the confounding effects of potential selection bias. We include more details about the community selection process is included in Annex II. DATA COLLECTION To measure the impact of the new schools on the outcomes of interest (e.g., primary completion rates, secondary school enrollment, reasons for dropping out of school, etc.), The evaluation team used quantitative and qualitative data collected in each community included in the study. NORC and its local partner Invest in Knowledge (IKI) created a panel of learners that were attending standards 5, 6, 7, and 8 at baseline and followed them over time. This included following all students in sampled schools who were in Standards 5-8 at baseline using administrative school data and tracking a sample of 20 students per standard per school interviewed at the 2018 baseline and at the 2021 endline, as well as tracked every year in between to update their enrollment status. Students in the tracked sample were randomly selected and chosen to include an equal number of boys and girls, wherever possible. A sample of caregivers was also randomly selected at baseline from a sub-sample of the tracked students. After baseline, follow-up data collection every year (2019, 2020 and 2021) included: (1) administrative school data to determine the current enrollment status of all students that were in Standard 5-8 at baseline; (2) student surveys administered to students from the tracked sample who have either dropped out or transitioned from primary school on to Form 1 since the 2018 baseline survey; (3) a survey administered to the caregivers of students in the tracked sample who have dropped out since the 2018 baseline; and (4) interviews with teachers and head teachers in newly constructed schools in the treatment communities. At endline (2021) we additionally included interviews with all students in our tracked sample, interviews with the mothers of tracked girls in our sample who had either dropped out or enrolled in secondary school, and interviews with mothers of girls in Standard 5-8. Table 3 presents details about the data collected in each of these surveys. Table 3: List of Data Collection Activities Instruments Main topics included Frequency Description Students 8 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Instruments Main topics included Frequency Description Student Interview Demographic information; travel time to/from school; household socio-economic characteristics information about learners’ older siblings; attitudes towards gender norms; learners’ aspirations and expectations about education, marriage, children, and work. Baseline (2018) 80 students per school, including 10 boys and 10 girls per grade. This group forms our tracked sample. Dropout Social and family context, reasons for Every Interview conducted with Interview dropping out exposure to violence, activities since dropping out of school follow-up year as appropriate (2019-2021) every tracked student who dropped out, in the first year they are observed as dropouts. Form 1 Social and family context, exposure Every Interview conducted with interview to violence, perceptions of gender norms follow-up year as appropriate (2019-2021) every tracked student who enrolled in secondary school, in the first year they are enrolled. Reinterview of dropouts and secondary students seen previously Update of school status, current activities, aspirations and expectations, attitudes towards gender norms Endline (2021) Re-interview of all tracked students who participated in the Dropout or Form 1 interview in either 2019 or 2020, conducted at endline. Caregivers Caregiver Attitudes regarding gender equity Baseline Caregivers of 40 students Interview and aspirations and expectations for their child. (2018) per school, including 5 boys and 5 girls per grade, selected from among the students in our tracked sample. Caregiver of Social and family context, reasons for Every Interview conducted with dropout dropping out exposure to violence, follow-up the caregivers of all Interview activities since dropping out of school year as appropriate (2019-2021) dropouts, in the first year they are observed as dropouts. Interview with female caregivers of girl dropouts and secondary students seen previously Knowledge and behavior related to sexual and reproductive health. Opinion about educating youth on sexual and reproductive health. Endline (2021) Interview of the female caregivers of tracked girls who participated in the Dropout or Form 1 interview in either 2019 or 2020, conducted at endline. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 9 Instruments Main topics included Frequency Description Interview with female caregivers of current Std. 5-8 girl students Knowledge and behavior related to sexual and reproductive health. Opinion about educating youth on sexual and reproductive health. Endline (2021) Interview with the caregivers of 20 girls per school at endline, 5 per each of Stds. 5-8. Selected independently of the tracked sample. Teachers and Community Leaders Teacher Interview Demographics, education, experience, tenure in new CDSS, satisfaction with CDSS, expectations about students. Every year since CDSS opened (2019-2021) Interview with every teacher in each CDSS. Teachers are re￾interviewed each year, including any new teachers. Head Teacher Interview Demographics, education, experience, tenure in new CDSS, satisfaction with CDSS, school fees. Every year since CDSS opened (2019-2021) Interview with the head teacher of each CDSS. Head teachers are re￾interviewed each year. Community leader Interview Attitudes regarding girls’ education School building process, satisfaction with new CDSS facilities Baseline and Endline (2018 & 2021) Interview with one leader per community, interviewed at baseline and endline. Focus Group Discussions Focus Group Discussions with girls and parents of girls (dropouts and secondary students), and with teachers and headteachers Barriers to girls’ education Satisfaction with CDSS facilities Endline SAMPLE SIZE The sample size for the tracked students was determined at baseline by the number of secondary schools being constructed (N=11). A mathematical calculation was used to estimate the smallest effect that could be detected with a sample of 11 treatment communities and an equal number of control communities, with 80 learners in each existing primary school, for a total sample of 1,760 learners.45 In other words, we ask what the smallest impact would be that we can detect with a reasonable level of confidence given this sample size. With this sample, we are able to detect a change in transition rates from primary to secondary school from 30% to 50% or higher when we do the analysis for each 4 We collected data from one additional comparison community. Initially the community was selected for treatment, however the construction site was not suitable and the community was rejected and replaced by another one. Since data from this community had already been collected, we decided to include it as part of the control group. 5 Sample calculation details in Annex II. 10 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA standard separately, and from 30% to 47% or higher when we analyze standards 6 to 8 altogether. In addition, we are able to detect changes from 50% to 70% or higher in primary school completion rates when analyzing standards separately, and from 50% to 65% or higher when analyzing all standards together. While school construction could generate impacts that are smaller than this, we cannot be reasonably confident that these smaller effects would show up as statistically significant findings in our estimations. Table 4. Tracked Learners Sample Treatment Communities Comparison Communities Baseline Interview Attritiona Status Attritionb Baseline Interview Attritiona Status Attritionb Schools 11 0 0 12 0 0 Standard 5 Learners 222 24 0 242 28 2 Male 112 10 0 115 8 1 Female 110 14 0 127 20 1 Standard 6 Learners 220 17 0 239 19 0 Male 105 7 0 119 11 0 Female 115 10 0 120 8 0 Standard 7 Learners 220 21 3 235 31 0 Male 109 13 1 117 15 0 Female 111 8 2 118 16 0 Standard 8 Learners 208 56 1 237 64 1 Male 109 29 1 112 25 0 Female 99 27 0 125 39 1 Total Learners 870 118 4 953 142 3 Male 435 59 2 463 59 1 Female 435 59 2 490 83 2 Note: Standard refers to standard at baseline. a Interview attrition refers to the number of students who could not be interviewed at endline. Reasons for not interviewing students at endline included refusals, inability to locate, or death.b Status attrition refers to the number of students whose grade level could not be determined at endline. Reasons for not interviewing students at endline included death, or inability to locate the student combined with no one at the student’s school being aware of their status. At the 2018 baseline, students were randomly sampled from primary schools in our treatment and comparison communities, selecting 10 male and 10 female students from each standard in each community, for a total of 80 students per community. Table 4 shows breakdown of the tracked students in our sample at baseline by standard and sex. The table also shows the levels of attrition at endline, both in terms of students who could not be interviewed at endline (interview attrition) and students whose grade level at endline could not be determined (status attrition). Levels of attrition were similar for treatment and comparison communities and for girls and boys. The level of status attrition was particularly low, at just 7 students out of the full sample of 1,823 (0.4 percent), while the level of interview attrition is higher and typical of a long-term panel study, with a total of 260 of 1,823 students (14.3 percent). Further details on the sample and sampling strategy are found in Annex II. ETHICS READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 11 For all data collection activities, NORC worked closely with its local partner, Invest in Knowledge Inc (IKI), a research and data collection firm based in Malawi. We obtained approval for this study and all associated data collection from both the NORC IRB and the National Committee on Research in the Social Sciences and Humanities, in Malawi. NORC and IKI worked in close collaboration with the District Education Officers to obtain all permissions and support for the data collection necessary for the evaluation. The NORC evaluation team provided training to all team members from the local data collection firm on confidentiality, anonymity and ethics, and special considerations when working with children and youth, including gender and violence training, and managing trauma and stress training. In addition, NORC uses a child protection protocol to follow during data collection from children on sensitive topics that has become the gold standard for USAID6. Based on a health assessment by counselors who accompany the data collection teams, child referrals are submitted to the community development and district social welfare officers as appropriate. LIMITATIONS While we have not identified any problems or limitations with the data collection work, there are some limitations inherent to the design of this evaluation. We list some of the more relevant limitations below: Representativeness of the Sample. The sample is representative of the areas where AMAA school construction took place, in the Machinga and Balaka districts. Therefore, results are not directly generalizable at the national level, other geographical areas, or to other school levels. Despite this, we hope the study will be informative and contribute to the evidence regarding secondary school construction, secondary school attendance, and dropouts in developing countries. Longitudinal Attrition. The evaluation employs a longitudinal design, where we interviewed the same students and teachers at baseline, follow-ups, and endline. Some students were still attending school, while others dropped out. Student status attrition was very low, as administrative data helped us identify student enrollment status and standard level. Interview attrition is quite reasonable for this type of panel data study, but it is present and should be kept in mind. School Sample Size. It should be noted that the nature of the activity necessarily limits our sample size. The sample is large enough to detect improvements in transition to secondary school from 30% to 50% and to detect changes from 50% to 70% in completion of primary school. Therefore, unless changes are large, estimates will tend to be imprecise. When this is the case, it is noted, and findings will not be taken as conclusive but rather suggestive of a particular trend. FINDINGS PRIMARY COMPLETION, ENROLLMENT IN SECONDARY SCHOOL, AND DROPOUT OUTCOMES 6 NORC, Child Protection Protocol for Data Collection: USAID/Uganda Performance and Impact Evaluation for Literacy Achievement and Retention Activity (Washington, DC: USAID, 2017) https://pdf.usaid.gov/pdf_docs/PA00XF22.pdf 12 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Table 5 shows a variety of outcome indicators for girls and boys in the tracked sample. The table shows that a higher percentage of girls than boys were dropouts at endline, a finding that is consistent regardless of standard at baseline7. Overall, 42.3 percent of all girls were dropouts at endline, compared to 34.9 percent of boys. Similarly, a lower percentage of girls (45.1 percent) than boys (51 percent) were enrolled in secondary school at endline; this finding is consistent across baseline standards, but only statistically significant overall. On the other hand, 35.2 percent of girls, overall, repeated any standard over the study period, compared to 39.4 percent of boys, a difference that is statistically significant. Among all girls, 50.5 percent had taken the Primary School Leaving Certificate Exam (PSLCE) and obtained a passing score, compared to 56.6 percent of boys. Table 5. Main Outcome Indicators, by Student Gender Overall Girls Boys Diff. Girls￾Outcome Percent Percent Percent Boys Student is a dropout at endline 38.7 42.3 34.9 7.5 *** Student was ever a dropout 39.7 43.3 36.0 7.3 *** Student is still in primary school at endline 25.6 24.1 27.0 -2.9 Student is in secondary school at endline 48.0 45.1 51.0 -5.9 ** Ever made it to secondary school 52.4 49.5 55.3 -5.8 ** Moved to Attend Secondary School 37.6 38.2 37.1 1.1 Student ever repeated a standard 37.3 35.2 39.4 -4.3 * Student ever took PSLCE Exam 61.4 60.0 62.8 -2.7 Student ever took PSLCE and obtained passing score 53.5 50.5 56.6 -6.2 *** *** p < 0.01, ** p < 0.05, * p < 0.1. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. The 1816 students include 462 in Standard 5 at baseline, 459 in Standard 6, 452 in Standard 7, and 443 in Standard 8. 1354 students had the potential to make it to secondary school by endline (i.e., were in Standard 6 or higher at baseline). We also show the percentage of students that were ever out of school, this percentage a slightly higher (one percentage point) than the percentage of dropouts at endline, showing that a very small proportion of students returned to school after a period out if it. Table 6 shows the same group of outcome indicators, this time comparing students in treatment and comparison communities, and including details by baseline standard. The table shows that for most baseline standard cohorts, the percentage of dropouts in the treatment group is lower than among the comparison sample. However, the differences are generally small and not statistically significant, except for students in Standard 7 at baseline. Overall, 38.9 percent of all students in the treatment sample were dropouts at endline, compared to 41.1 percent for the comparison group. Similarly, the percentage of students that were out of school at any point since the baseline is higher in the comparison than in the treatment communities, but the difference is not significant. The data also suggests that in treatment areas, students take longer to complete primary school than in comparison areas. The percentage of students still in primary school at endline is higher (26.1 vs. 23.2 in treatment and comparison areas, respectively), and there is more repetition (40.2 vs. 34.7). According to the teachers, students sometimes choose to repeat standards as part of a strategy to maximize the probability of obtaining higher marks on the PSLCE and qualify to attend the best possible public school. In our sample, fewer 7 See details of these findings by student baseline standard in Annex IV. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 13 treatment than comparison learners have already taken the PSLCE, but their exam pass rate is slightly higher (53.6 vs 52.9), although none of the differences are statistically significant. Table 6: Main Outcome Indicators, by Treatment Status Treatment Comparison Diff. Treat￾Pct. Pct. Comp Student is a dropout at endline 38.9 41.1 -2.2 Standard 5 45.9 39.8 6.1 Standard 6 41.5 42.7 -1.2 Standard 7 37.5 45.9 -8.4 * Standard 8 30.2 35.8 -5.5 Student was ever a dropout 39.8 42.4 -2.5 Standard 5 47.2 42.1 5.2 Standard 6 41.9 44.0 -2.0 Standard 7 38.0 45.9 -7.9 * Standard 8 31.7 37.3 -5.6 Student is still in primary school at endline 26.1 23.2 2.8 Standard 5 54.1 60.2 -6.1 Standard 6 38.7 27.7 11.0 ** Standard 7 9.3 4.0 5.3 ** Standard 8 0.5 0.0 0.5 Student is in secondary school at endline 47.0 47.7 -0.7 Standard 5 n/a n/a n/a Standard 6 19.8 29.7 -9.9 ** Standard 7 53.2 50.1 3.1 Standard 8 69.3 64.2 5.0 Ever made it to secondary school 50.5 52.8 -2.3 Standard 5 n/a n/a n/a Standard 6 19.8 29.7 -9.9 ** Standard 7 56.0 54.4 1.6 Standard 8 77.1 75.6 1.5 Student ever repeated a standard 40.2 34.7 5.5 ** Standard 5 52.3 38.8 13.4 *** Standard 6 49.3 41.6 7.7 Standard 7 38.9 32.7 6.2 Standard 8 19.0 25.0 -6.0 Student ever took PSLCE Exam 60.9 61.5 -0.6 Standard 5 17.9 26.5 -8.7 ** Standard 6 52.5 53.9 -1.4 Standard 7 78.2 69.3 8.9 ** Standard 8 97.1 98.0 -0.9 Student ever took PSLCE and obtained passing score 53.6 52.9 0.7 Standard 5 13.8 22.2 -8.5 ** Standard 6 43.3 47.0 -3.6 Standard 7 69.4 59.0 10.5 ** Standard 8 90.2 85.0 5.3 * *** p < 0.01, ** p < 0.05, * p < 0.1. All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. The 1816 students include 462 in Standard 5 at baseline, 459 in Standard 6, 452 in Standard 7, and 443 in Standard 8. 1354 students had the potential to make it to secondary school by endline (i.e., were in Standard 6 or higher at baseline). 14 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Table 7 shows the total percentage of students that moved to attend secondary school and the distributions for treatment and comparison communities. While almost 29 percent of the students in treatment communities moved, the percentage of students in comparison communities was significantly higher, reaching over 46 percent. The decision to move to attend secondary school may be driven by the lack of a secondary school within reach, family migration, or by being able to attend a National, District or other secondary school for the students with highest PSLCE. According to the Education Sector Analysis prepared in 2019 by the MoEST, students with the highest PSLCE are selected to enroll in one of the four National Secondary Schools. In addition, there are ten grant-aided secondary schools where some selected students may enroll. Remaining selected students are sent to the district secondary schools according to the location of the primary school. The next level of PSLCE successful graduates are placed in the boarding and day secondary schools, which represent the second-tier level of about 130 schools. The remainder are selected to about 700 CDSSs, which are community-owned and developed in areas where there are no other secondary schools. These CDSSs are funded by the communities, with teacher salaries paid by the formal system national payroll. AMAA CDSSs are part of this group of secondary schools. Table 7: Moving to Attend Secondary School Overall Treatment Comparison Diff. Comp￾Treat Pct. Pct. Pct. Moved to Attend Secondary School 36.7 28.7 46.2 -17.5 *** *** p < 0.01, ** p < 0.05, * p < 0.1. All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. Table 8 shows that, among students enrolled in secondary school, more are enrolled in public schools in treatment communities (92.4 percent) than in comparison communities (81.5 percent). Not surprisingly, in treatment communities most secondary students are enrolled in an AMAA CDSS (69.3 percent), and slightly over 20 percent are enrolled in other public schools (National, District, etc. secondary schools). Almost 21 percent of the students from comparison communities are also enrolled in AMAA CDSSs, showing that AMAA not only benefitted local children and youth but also students from neighboring areas, including our comparison population. Table 8: Secondary School Enrollment – School Type Secondary School Type Comparison Pct. (%) Treatment Pct. (%) Diff. Comp-Treat AMAA CDSS 20.8 69.3 48.5*** Other Public School 60.7 23.1 -37.6*** Private School 18.5 7.6 -11.0*** Observations 286 252 n/a Note: All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics. Table includes only students enrolled in secondary school before 2021. There were 650 students enrolled in secondary before 2021, and 538 could be located and responded to the question at endline. *** p < 0.01, ** p < 0.05, * p < 0.1 READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 15 DROPOUT LEARNERS8 Many learners (around 33 percent of boys and 32 percent of girls) that dropped out of school in our sample, did so after taking the PSLCE and did not start secondary school, as shown in Table 9. Slightly more than 22 percent of girls and 23 percent of boys left school after completing Standard 6 and a similar percentage after completing Standard 7. Fewer learners dropped out after completing Form 1, 9.3 and 7.7 percent for boys and girls, respectively. Between five and six percent left school with only Standard 5 completed. Table 9: Completed Level of Education before Dropping Out of School Enrollment Level of Education Before Dropping Out Boys Pct. (%) Girls Pct. (%) Diff. Boys-Girls Standard 5 5.1 5.9 -0.8 Standard 6 22.1 23.6 -1.5 Standard 7 22.1 23.3 -1.2 Standard 8 – Incomplete 8.0 7.4 0.6 Standard 8 – Took PSLCE 33.3 32.1 1.3 Form 1 9.3 7.7 1.6 Observations 312 390 n/a Note: Table includes students who were not enrolled in school at endline. 702 students met this definition, and highest education could be determined for 702. *** p < 0.01, ** p < 0.05, * p < 0.1 At baseline we collected detailed information about students’ sociodemographic characteristics. We use that information to study which factors are associated with dropping out. Table 10 shows the results of a series of regressions that predict the association of student characteristics with the probability of dropping out. We find that, after controlling for a student’s standard, age is positively associated with dropping out. In other words, at any given standard older children are more likely to abandon school. Specifically, a one-percentage point increase in age is associated with an increase of approximately 1.5 percentage points in the risk of dropout; for example, a student who is aged 11 at baseline (a ten-percent increase over age 10), would be expected to have a 15-percentage point increase in the risk of dropout, compared to a student aged 10. The effect is important, given that most students are older than the appropriate age for their grade level, with just 24.6 percent of female students and 17.4 percent of male students on-track given their age at baseline. Many students were significantly behind their expected grade level for their age, with 26 percent of females and 40 percent of males two or more years behind at baseline. While the expected age for a student in Standard 8 is 14, 22 percent of students in Standard 8 were 18 or older at baseline. As children make progress through standards in primary school, they are less likely to dropout. For example, a student in Standard 6 at baseline is approximately 8 percentage points less likely to dropout, after controlling for age and other traits, than a student in Standard 5, while a student in Standard 8 is approximately 40 percentage points less likely to dropout. Being a girl is associated with higher probably of dropping out. We estimate that other things equal, a girl student is approximately 14 percentage points more likely to dropout than a boy student in our sample. The difference is statistically significant 8 See Menendez et al., 2022 “Barriers to Girls’ Education in rural Malawi. Evidence from Machinga and Balaka districts” for an in-depth study about dropouts. 16 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA and large in magnitude. Students that reported more school absenteeism at baseline and those that lived further from the school were also more likely to leave school, although distance from school is not statistically significant. Living with a mother that finished primary school is associated with a decreased risk of dropping out. The same is true for a socioeconomic score based on home assets possession, however the relationship is not statistically significant in our sample. Finally, having a new CDSS in the community is negatively related to the probability of dropout, but the estimate is not statistically significant, and we cannot conclude that is different from zero. Table 10: Factors associated with school dropout. Multivariate regressions Robust standard errors in parentheses, clustered at the school-cohort level. All regressions conducted using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics a Socioeconomic score calculated as the percentage of items owned in a list of 13 common household items. *** p<0.01, ** p<0.05, * p<0.1 During individual interviews with children that dropped out of school, we asked the reasons behind their dropout decisions. Table 11 shows the most common reasons given. By far, the most common reason was lack of financial resources to afford the cost of education. This is a problem faced by both boys and girls, although it was most commonly mentioned by boys. The second most common reason for leaving school was pregnancy. More than a quarter of the girl dropouts reported pregnancy as the reason for abandoning their schooling. This fraction increases to a third of the dropout girls if we also consider marriage and having/nursing a baby. Overage girl students (i.e., students that are older than the expected age for their standard) are the most likely to dropout. Among girls who drop out and have (1) (2) (3) (4) VARIABLES Dropout Dropout Dropout Dropout Ln (Age at baseline) 1.57*** 1.54*** 1.53*** 1.51*** (0.09) (0.08) (0.09) (0.09) Standard 2018 = 6 -0.08** -0.08*** -0.07** -0.08** (0.03) (0.03) (0.03) (0.03) Standard 2018 = 7 -0.19*** -0.19*** -0.19*** -0.19*** (0.04) (0.03) (0.04) (0.04) Standard 2018 = 8 -0.40*** -0.40*** -0.41*** -0.41*** (0.04) (0.04) (0.05) (0.04) Student is a girl 0.14*** 0.14*** 0.14*** 0.14*** (0.02) (0.02) (0.02) (0.02) Missed any school, week before (at baseline) 0.06** 0.05* 0.06** 0.05* (0.02) (0.03) (0.02) (0.03) Ln (minutes get to school at baseline) 0.01 0.02 0.01 0.02 (0.01) (0.01) (0.01) (0.01) Socioeconomic scorea -0.10 -0.09 -0.08 -0.08 (0.08) (0.07) (0.08) (0.07) Living (at baseline) with a mother who completed primary school -0.07*** -0.06** (0.02) (0.02) AMAA CDSS open in student’s community -0.04 -0.06 (0.03) (0.06) Constant -3.72*** -3.67*** -3.58*** -3.55*** (0.23) (0.22) (0.23) (0.230) Observations 1,776 1,776 1,776 1,776 R-squared 0.16 0.18 0.16 0.19 School FE No Yes No Yes READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 17 ever been pregnant, around 65 percent had their first pregnancy at age 18 or older. Not surprisingly, this problem does not affect males. Just 1.5 percent of boys who dropped out listed their partner getting pregnant as their reason for dropping out. No motivation for schooling and not doing well in school were mentioned by boys and girls in 17.2 percent of the cases, when taken together. Long distances from home to school was cited as a reason for dropping out in 6.2 percent of the cases, similar for boys and girls. Problems with teachers were mentioned by 5.5 percent of the boys, and by 2 percent of the girls. Other reasons such as illness or need to work were mentioned, although not frequently, and very few children (1.3 percent) mentioned their parents being opposed to their education. Table 11: Reasons for dropping out of school self-reported by learners. Reason Overall % Boys % Girls % Diff. Boys￾Girls Treat ment % Compa rison % Diff. Treat￾Comp Fees/Financial/Costs 61.4 69.7 54.8 15.0*** 60.7 62.1 -1.4 Pregnancy 15.9 1.5 27.2 -25.8*** 14.1 17.7 -3.6 Do not like school 9.1 9.6 8.7 0.9 9.2 9 0.2 Not doing well at school 8.1 9.6 7.0 2.6 10.2 6.1 4.1* Travel/distance to school 6.2 6.3 6.1 0.2 4.6 7.7 -3.1 Marriage 4.1 3.0 4.9 -2.0 4.6 3.5 1.1 Problems with/afraid of teacher 3.6 5.5 2.0 3.5** 3.6 3.5 0.1 Need to work/earn money 2.3 4.4 0.6 3.8*** 3 1.9 1 Respondent sick/ill 2.4 2.2 2.6 -0.4 3 1.6 1.3 Not selected to secondary school 1.5 2.2 0.9 1.3 2.3 0.6 1.7* Parents/guardian do not want 1.3 1.8 0.9 1.0 1 1.6 -0.6 Death of HH member 1.3 1.5 1.2 0.3 1.6 1 0.7 Illness of HH member 0.8 0.7 0.9 -0.1 1.3 0.3 1 Has baby/nursing baby 0.6 0.0 1.2 -1.2* 1.3 0 1.3** Problems with other students 0.6 0.7 0.6 0.2 0.7 0.6 0 Need to look after siblings 0.5 0.7 0.3 0.4 0.7 0.3 0.3 Need to work at home / fields 0.2 0.0 0.3 -0.3 0 0.3 -0.3 Note: This question was a multiple response question, so counts for individual items may not add up to the total number of respondents. *** p < 0.01, ** p < 0.05, * p < 0.1. Table includes students who ever dropped out between 2018-2021. 721 students met this definition, 632 were successfully tracked, and 616 of these responded to the survey question. All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics In general, the reasons to leave school that were discussed in the focus groups overlap with those given by boys and girls during the quantitative interviews. 18 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA School fees and education costs were the most cited barrier to attend schools by girls and boys in treatment and comparison communities. Education is costly: the average enrollment fees per term are around MWK 1,850 for public primary schools, and for public secondary schools the average is MWK 14,1509 and may reach higher levels. Teachers and students reported that it is school policy to send home the students who have not paid fees. “When we tell our parents about school fees, they tell us that they do not have fees. When we come to school, they send us back. We stay home for about a week or two waiting for the school fees. When we come back to school, we find that our friends have covered a lot of things…” (Girl attending school, Kayuni) The average fees in AMAA CDSSs are MWK 10,918 per term. Table 12 shows the proportion of secondary learners in our sample that receive a bursary to cover school fees in AMAA and in other schools. The support for students in AMAA schools is slightly more than double than in other schools, reaching 30 percent. The students receiving bursaries in AMAA schools receive, on average, MWK 9,665 per term. Table 12: Percentage of secondary learners receiving support to pay school fees Are you receiving any monetary help, such a bursary, scholarship, or other to pay the school fees? AMAA CDSS Other school Pct. (%) Pct. (%) Diff. Yes 30.1 14.2 15.9*** Observations 143 395 Note: *** p < 0.01, ** p < 0.05, * p < 0.1 At Muthe, one of the comparison communities, the secondary school is located far away from the village, so families need to pay for boarding and associated costs, in addition to the school fees. In contrast, in the communities where the new CDSSs were built, families can save on boarding costs. As with enrollment fees, families may struggle to afford school learning materials and other educational necessities. Students who arrive at school without their uniforms are sent away per school policies. If students only have one uniform, then they must wash it at night, and sometimes they lack soap. “Lack of soap to wash clothes also contributes to the drop out. They prefer to stay at home than to go to school with dirty clothes.” (Parents of girls attending school, Nambaiko) Additionally, lack educational materials, including notebooks, schoolbags, exercise books, pens/pencils, and calculators, deters children from attending school. Table 13 shows the proportion of students that received monetary support to cover education expenses other than fees. Very few students benefit from this kind of support. The percentage is two percentage points higher in AMAA schools than in others (8.7 vs. 6.7 percent, respectively) but the difference is not statistically significant. 9 MWK 14150 were approximately USD 17 at the time of the endline in December 2021 while the Malawi GDP per capita in 2021 was estimated in USD 567 (IMF https://www.imf.org/en/Countries/MWI#countrydata). READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 19 Table 13: Percentage of secondary learners receiving support to pay school expenses other than fees Do you receive any other help to cover expenses on books, supplies, uniforms, or other items? AMAA CDSS Other Schools Pct. (%) Pct. (%) Diff. Yes 8.7 6.7 2.0 Observations 143 399 Note: Table includes only learners enrolled in secondary school in 2021. There were 650 learners enrolled in secondary, and 542 could be located and responded. *** p < 0.01, ** p < 0.05, * p < 0.1 POST-DROPOUT ACTIVITIES We asked dropouts how they spend their time. Table 14 shows the findings for girls and boys. Among girls, the most common activity is working in the family farm or business without pay (39.7 percent), followed by doing nothing (25.8 percent) and caring for their child (16.5 percent). In the case of boys, it is most common to work for pay (34.3 percent), followed by working in the family farm or business without pay (30.6 percent) and doing nothing (18.8 percent). Around 5 percent of girls and boys report that they spend their time hanging around with friends. Across communities, girls who have dropped out are more likely to be involved in daily household chores and caring for others than boys (3.5 vs. 0.4 percent for girls and boys, respectively). Boys are more likely to attend vocational training or apprenticeships (4.4 percent for boys and 1.4 percent for girls). Table 14: Activities among dropouts Girls Boys After you left school, what have you done most of the time? Pct. (%) Pct. (%) Difference I have been working in a family business/farm without pay 39.7 30.6 -9.1** I have been working for pay 6.4 34.3 27.9*** Nothing 25.8 18.8 -7.0** I have been caring for my child 16.5 0.4 -16.2*** I have been looking for a job 0.9 5.9 5.0*** I have been hanging around with friends 5.2 4.8 -0.4 I have been caring for child or adult in my family 3.5 0.4 -3.1*** I have attended vocational training/apprenticeship 1.4 4.4 3.0** I have been traveling 0.6 0.4 -0.2 I have been ill or disabled and unable to work 0.0 0.0 0.0 Do not Know 0.0 0.0 0.0 Refuse 0.0 0.0 0.0 Observations 345 271 *** p < 0.01, ** p < 0.05, * p < 0.1 During FGDs girls confirmed the quantitative findings. When there is seasonal or part-time work available, such as fetching water for neighbors, providing childcare for others, or selling harvests, some 20 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA girls will take these opportunities to financially support their families. However, the lack of job opportunities in their communities is often a reason why they cannot work beyond the occasional odd job. Among those that work, almost 70 percent work at their family dwelling, garden, or farm, followed by ten percent that works at a street or market stall and eight percent that does not have a fix place of work. Half of the working youth reports being unsatisfied or very unsatisfied with their job, eight percent feels neutral about it and 42 percent is satisfied or very satisfied. EXPECTATIONS AND ASPIRATIONS FOR DROPOUTS AND LEARNERS At baseline we asked students about their aspirations and expectations regarding their education10. We collected the same information from caregivers and asked community leaders how they see the future of the youth in their villages. In general, aspirations and expectations from parents, children and local leaders were very high. This was not totally surprising. Though research on parents’ and children’s educational aspirations in developing countries is sparse, the existing literature shows that expectations about education are consistently much higher than actual educational attainment results (Guerero et al., 2016; Tafere, 2014; Adams et al., 2014; Chowa, 2007; Beutel, 2007). In Table 15 we show the differences in aspirations and expectations that children that dropped out and children still enrolled in school had at baseline. Considering the context, the educational aspirations and expectations were very high for both groups, although children and youth still enrolled in school show even higher aspirations and expectations than dropouts. For example, almost 75 percent of students expected to finish college or university studies, while the percentage among those that dropped out was slightly over 68 percent. Those still enrolled in school were also three percentage points more likely to say that they expected to have a job by age 25. Although we observe these earlier differences, it is important to keep in mind that at baseline all students expressed exceedingly high educational aspirations and expected them to be achieved. In addition, we found no differences between girls and boys. Table 15: Aspirations and expectations at baseline, students, and dropouts Aspirations and Expectations Enrollment Status at Endline Diff. Dropout -Enrolled Dropout (Mean) Enrolled (Mean) Aspirations and Expectations at Baseline Highest desired level of education is college/university 82.6% 87.9% -5.4*** Expects to finish college/university 68.3% 74.3% -6.1*** Age student expects to complete their formal education 24.5 23.8 0.7*** Student would like to have a job by age 25 91.4% 94.3% -2.9** Expects to be able to work in their preferred job by age 25 82.1% 84.8 -2.7 Student would like to get married at some point in their life 87.1% 89.5% -2.4 10 We adapted our questions from the longitudinal Young Lives Survey (https://www.younglives.org.uk/) which has been successful at collecting this type of data in several developing countries. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 21 Aspirations and Expectations Enrollment Status at Endline Diff. Dropout -Enrolled Dropout (Mean) Enrolled (Mean) Approximate age student would like to get married 28.6 28.4 0.2 Student would like to have children at some point in their life 88.0% 90.9% -2.9** Approximate age student would like to have their first child 30.5 30.6 -0.1 On a scale of 1-9, where 1 is the worst possible life and 9 is the best possible where do you stand at the present time? 3.2 3.4 -0.2 On a scale of 1-9, where 1 is the worst possible life and 9 is the best possible life, where do you think you will be at age 25? 7.0 7.1 -0.1 Total Observations 702 1,114 Aspirations and Expectations at Endline Where on the ladder do you feel you personally stand at the present time? 3.9 3.9 0.1 Where on the ladder do you think you will be in the next 5 years? 6.2 6.4 -0.2 Total Observations 397 820 *** p < 0.01, ** p < 0.05, * p < 0.1 Expectations about marriage and marriage age were not significantly different between the groups, although those still in school at endline were more likely at baseline to say they wanted to have children. In general, students expected to continue their education well into their twenties; on average, they expected to marry after the age of 28, and only have children after turning 30. This created a ceiling effect, where given such high baseline expectations, there was little room for the newly built AMAA schools to raise expectations any further. At baseline, we also explored students’ expectations for their quality of life in adulthood. We asked the students about their position at the time of the interview and at age 25 on a nine-step quality of life ladder, where the first step represented the worst possible life, and the ninth step represented the best possible life. In general, students were optimistic about their future. They located themselves just above step 3 at baseline but expected to be in the highest half of the ladder at age 25. There were no differences between those that later dropped out and those that remained enrolled in school. At endline we asked similar questions about quality of life, and we found no differences between secondary students and dropouts. At the time of the interview, both groups located themselves around step 3.9, slightly higher than at baseline. Dropouts expected to be at step 6.2 in five years, while secondary students expected to be slightly higher, at step 6.4. This is a very minimal difference that suggests dropouts do not feel they are in worse situation than their counterparts still in school and that their expectations about the future do not seem affected by the fact they had left school. We did not find differences between treatment and comparison communities. When we compare aspirations and expectations between baseline and endline (Table 16), we observe that an even larger fraction of students than at baseline wants to attend college/university and there are no differences between treatment and comparison communities. However, the expectations were adjusted down from baseline to endline and fewer students think they will be able to reach their desired 22 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA education levels. The adjustment is more pronounced among those already in secondary school than students still in primary school. Students in treatment communities do not show higher expectations than their peers in comparison communities. Table 16: Aspirations and expectations at baseline and endline in treatment and comparison communities Mean Value Diff. Endline￾Baseline Baseline 2018 Endline 2021 Highest desired level of education is college/university Treatment 87.0% 90.2% 3.2 Still in primary at endline 80.7% 85.4% 4.7 In secondary at endline 93.7% 97.3% 3.6 Comparison 84.4% 89.1% 4.6* Still in primary at endline 81.5% 87.0% 5.5 In secondary at endline 91.6% 100.0% 8.4* Expects to reach desired level of education Treatment 67.1% 54.0% -13.1*** Still in primary at endline 71.3% 62.7% -8.6* In secondary at endline 65.0% 40.8% -24.2*** Comparison 68.7% 62.6% -6.2* Still in primary at endline 74.0% 64.8% -9.2** In secondary at endline 66.3% 50.5% -15.8** Finally, we also estimated whether the newly built AMAA schools affected aspirations and expectations using a Difference in Difference (DiD) approach. This approach compares if changes from baseline to endline have been different between children in the communities where the AMAA CDSSs were built and children in the comparison communities.11 We did not find any difference between AMAA and comparison groups. The details of the estimations can be found in Annex IV. PERCEPTIONS AND EXPERIENCES WITH VIOLENCE FOR DROPOUTS AND STUDENTS SCHOOL CLIMATE We read secondary level students a series of statements about the climate in their schools and asked them to rate the degree to which they agreed or disagreed. Table 17 shows the percentage of students that agreed or strongly agreed with each statement, comparing students in AMAA schools to those enrolled in other secondary schools. In general, the students’ responses imply a safer climate in AMAA schools. The students in AMAA schools report higher perception of safety and more parental and 11 We conducted the analysis for all communities pooled together and also for the Balaka communities separately given that at baseline the treatment communities in Machinga had been selected and that could have affected baseline aspirations and expectations in anticipation of the CDSSs construction. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 23 community involvement in their schools. Although the differences are not always statistically significant, considering all the findings together seems to suggest that AMAA CDSSs are doing a better job than other secondary schools offering a positive school climate. Table 17: Secondary school climate according to students AMAA CDSS Other Schools Diff. Agree % Agree % Sig I feel safe at my school 74.6 71.7 3.0 My friends are safe within this school 69.7 63.5 6.2 The school is a welcoming place for my parents 83.5 80.3 3.1 My parents encourage me to take school seriously 99.3 95.2 4.1 ** Adults in the community know what goes on inside this school 60.6 54.8 5.8 Families are involved in decisions that protect children at this school 64.1 65.3 -1.2 Sometimes children are unsafe when traveling to and from this school 38.5 41.4 -3.0 Violence is a major concern within this school 35.3 42.5 -7.2 Students are afraid to come to this school 5.7 12.0 -6.3 ** Observations 138 522 *** p < 0.01, ** p < 0.05, * p < 0.1. Table includes students in the sample who enrolled in secondary school. 709 students met this definition, and 660 could be successfully interviewed and responded to these questions. The survey included questions about witnessing violence. We asked secondary school students and dropouts about witnessing violence by adults in the community and by a caregiver (e.g., against a spouse, partner, siblings). Table 18 shows the percentage of children that responded affirmatively. While witnessing violence by adults in the community is 8.6 percentage points more common among dropouts than secondary students, witnessing caregiver physical violence at home is more almost 8 percent points more common among children enrolled in school at endline. Table 18: Percentage of Students and Dropouts that Witnessed Violence in the Community and at Home Enrollment Status at Endline Diff. Dropout￾Enrolled Dropout % In Secondary % Witnessed community adult violence 77.3% 68.7% 8.6 pp ** Witnessed caregiver physical violence 38.8% 46.6% -7.8 pp ** Observations 510 659 *** p<.001, ** p<.05, * p<.1 Finally, we asked secondary school students and children that dropped out of school about their own experiences being victims of violence. The survey included a battery of questions on different forms of emotional, physical, and sexual violence. To summarize answers to these questions for each type of violence, we created binary variables that take the value of 1 if the respondent disclosed any instance of 24 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA violence victimization, and 0 otherwise.12 Table 19 shows the percentage of dropout and secondary school children that have been victims of these types of violence. Around a third of the respondents reported being a victim of emotional violence, with no significant difference between dropouts and secondary students. Slightly more than half -around 54 percent- of dropouts and students reported being victim of physical violence from an adult or a peer. Almost 38 percent of children that left school reported having been victims of sexual violence. The percentage is 8.6 percentage points lower among secondary students (28.3 percent); the difference between the two groups is statistically significant. Table 19: Prevalence of Emotional, Physical and Sexual Violence at Endline, Students and Dropouts Enrollment Status at Endline Diff. Dropout￾Enrolled Dropout % In Secondary % Emotional Violence 31.0% 31.3% -0.3 pp Physical Violence by Adult or Peer 53.9% 54.2% -0.3 pp Sexual Violence 33.5% 28.3% 5.3** Observations 510 659 *** p<.001, ** p<.05, * p<.1 TEACHERS IN CDSSS SCHOOLS. PROFILE AND RETENTION To analyze teacher recruitment and retention, we conducted a survey of teachers in the 11 AMAA CDSSs. As explained before, five schools in Machinga started operating in the 2018-2019 school year, while six schools in Balaka started a year later. We surveyed all teachers and head teachers in these schools in 2019, 2020 and 2021. The target number of teachers in CDSSs is eight, including the head teacher. Figure 2 shows the average number of teachers (including the head teachers) in each AMAA school, by year and district. In 2019, the five Machinga schools had on average 6.6 teachers. In 2020, Machinga’s schools retained 5.4 teachers and recruited, on average, three new teachers, reaching an average of 8.4 teachers per school. Also in 2020, schools in Balaka recruited 6.2 teachers, on average. In 2021, schools in Machinga retained 7.6 teachers and recruited 1.6 new teachers, on average, while Balaka schools on average retained 4.7 teachers and recruited 1 new teacher. On average, schools have been able to attract enough teachers to substitute the ones that left. Schools started with fewer than eight teachers, but in the case of Machinga the average number of teachers per school has increased every year, reaching the expected eight. The AMAA schools in Balaka maintained a constant average of 6 teachers per school. 12 Annex III lists all the questions used to construct each summary variable. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 25 Figure 2: Average number of teachers per school Table 20 shows descriptive statistics by teacher cohort. Cohort 1 corresponds to teachers that started working in the 2018-2019 school year, cohort 2 corresponds to teachers that started working in 2019- 2020, and cohort 3 corresponds to teachers that started working in the 2020-2021. Roughly 20 percent of teachers were females in cohorts 1 and 2, and 35.7 percent were females in cohort 3. In the 2019 cohort, 28.9 percent had a Form 4 education with certificate, 15.8 percent had a diploma in education, and 55.3 percent had a degree in education. Teachers’ education levels are lower in subsequent cohorts, as the fraction of teachers with a Form 4 certificate and diploma increased, while the fraction of teachers with a degree in education decreased. Age and experience also declined throughout cohorts. These results indicate that teacher qualifications have declined over time. Teachers in cohort 3 have less education and have less experience than teachers from cohort 1. This could suggest difficulties to attract high quality teachers to work in these rural CDSSs. This not a problem exclusive to the AMAA CDSSs. According to the MOEST Education Sector Analysis (2019) “secondary schools have many unqualified teachers (about 60%), with 81% of teachers unqualified in Community Day Secondary Schools (CDSS) and 27% in non-CDSSs.” Table 20: Descriptive statistics by teacher cohort Teacher Demographics First Observed Survey Year 2019 2020 2021 Head Teacher (%) 13.2 10.3 0.0 Female (%) 21.1 19.0 35.7 Highest Education (%) Form 4 with certificate 28.9 34.5 42.9 Diploma 15.8 34.5 50.0 Degree 55.3 29.3 7.1 Age (Average) 38.3 35.6 34.0 Years Teaching Overall (Average) 12.5 10.8 9.1 Number of forms (1-4) taught (Average) 2.6 1.6 1.8 5.4 7.6 4.7 6.6 3.0 6.2 1.6 1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 Machinga Machinga Balaka Machinga Balaka 2019 2020 2021 Old New 26 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Teacher Demographics First Observed Survey Year 2019 2020 2021 Lives in housing provided by school (%) 5.4 27.8 21.4 Lives in this community (%) 40.5 48.1 64.3 Does not live in community (%) 59.5 51.9 35.7 Distance to come in, in Minutes (Average)* 37.9 34.8 41.2 Days unable to come to school (Average) 0.6 1.1 0.4 N 38 58 14 Note: Data for five teachers not surveyed in their first year is imputed using subsequent years, except for questions on residence and attendance which are left missing for these teachers. *Only teachers living in a community other than the schools. Table 21 shows retention rates for teachers by teacher cohort. Out of the 38 cohort 1 teachers, 32 (84 percent) were retained in 2020, and 30 (79 percent) were retained in 2021. In 2020 out of the 58 cohort 2 teachers 47 (81 percent) were retained in 2021. Table 21: Retention rates by teacher cohort Retained teachers 2020 2021 Cohort 1 (N=38) 32 (84%) 30 (79%) Cohort 2 (N=58) N/A 47 (81%) Cohort 3 (N=14) N/A N/A The pooled one-year retention rate for cohorts 1 and 2 is 82.3 percent, for an attrition rate of 17.7 percent. Mulkeen (2010) estimates that the teacher attrition rate for secondary school teachers in Malawi is 10 percent, so the results for the AMAA CDSSs seem high. However, it is important to keep in mind that 17.7 percent corresponds to first-year retention and these schools are located in rural areas. It is natural to expect that attrition in new schools is higher the first few years than in the long run. At the beginning teachers assess whether the job conditions and other contextual factors correspond to their needs and preferences, so attrition may be high, but with time attrition should fall as teachers that are a good fit for the school decide to stay. Although we only have three years of data, already in year two attrition is much lower than year one: out of the cohort 1 teachers the retention rate between 2020 to 2021 is 93.8 percent (30/32). Along these lines, although one-year attrition seems high in the AMAA CDSSs, after two years retention seems comparable to previous estimates for Malawi. To analyze factors associated with retention, Table 22 shows sociodemographic statistics for teachers, comparing those that left the school after one year to teachers who were retained for at least one additional year. The table includes teachers from cohorts 1 and 2. All head teachers were retained. Twenty-two percent of retained teachers are females, while only 12 percent of teachers that left are females, which indicates that females are more likely to stay. Education is positively correlated with teacher turnover, as teachers who left after one year are more likely to have a degree than retained teachers. There are no major differences for age or experience between leaving and retained teachers. Roughly 18 percent of teachers live in housing provided by the school, regardless of their retention status. Leaving teachers are 20 percentage points more likely to live outside the school’s community than retained teachers. Furthermore, among teachers that live outside the school’s community, leaving teachers take longer to get to the school than retained teachers. In terms of teacher/student ratio, or READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 27 the number of forms teachers teach, there are no major differences between leaving and retained teachers. Table 22: Sociodemographic characteristics by retention status Teacher Demographics Observed following year Left Retained Total Count Head Teacher (%) 0.0 13.9 96 Female (%) 11.8 21.5 96 Highest Education (%) Form 4 with certificate 35.3 31.6 96 Diploma 11.8 30.4 96 Degree 52.9 36.7 96 Age (Average) 36.7 36.7 96 Years Teaching Overall (Average) 10.8 11.6 96 Number of forms (1-4) taught (Average) 2.2 2.0 96 Residence (%) Lives in housing provided by school 17.6 18.9 91 Lives in this community 29.4 48.6 91 Does not live in community 70.6 51.4 91 Distance to come in, in Minutes (Average)* 48.3 32.3 50 Days unable to come to school (Average) 1.2 0.8 91 Student Teacher Ratio at School (Average) 18.3 19.1 96 Observations 17 79 Note: Data for five teachers not surveyed in their first year is imputed using subsequent years, except for questions on residence and attendance which are left missing for these teachers. *Only teachers living in a community other than the schools. STATISFACTION WITH SCHOOL FACILITIES We discussed with girls, their caregivers, teachers and head teachers and community leaders their satisfaction with the building facilities of the new CDSSs. In general, beneficiaries reported that they were satisfied with the modern aesthetics of the new CDSSs building facilities. Crucially, stakeholders citied that by cutting down on students’ travel time to school, the CDSSs increased the ease of attending school for students in the AMAA communities. Stakeholders also explained that schools promoted student learning by providing some good classroom essentials (e.g. desks, chairs, boards). However, stakeholders were disappointed by a lack of several features in the CDSSs. The most commonly cited issues included teacher housing units, electricity, fences to clearly demarcate school boundaries, and instructional materials such as textbooks and lab equipment. These unmet expectations affected teacher satisfaction and possibly retention, perceived safety on school grounds, and student learning due to staffing shortages and disruptions during class hours. Below we describe the aspects that stakeholders appreciate the most and the issues they identify as challenging. 28 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA OVERALL STRENGTHS CDSS COMMUNITY PROXIMITY Community leaders, speaking on behalf of their perceptions of the community at-large, generally agreed that the students have benefitted from the construction of new CDSSs closer to their homes. Across all communities, respondents mentioned that students would often drop out or opt to marry if they could not walk to their school or if families were not able to cover boarding and transportation costs. These testimonies suggest that by making it easy for a student to physically travel to school, and removing the need for families to cover additional transportation and boarding expenses, the CDSSs removed a primary barrier to secondary school enrollment. Stakeholders in Njerenje and Kayuni reported that the construction of the new CDSS led to an increase in student enrollment. CLASSROOM INFRASTRUCTURE AND MATERIALS All communities reported a sense of satisfaction with the CDSS appearance and construction, describing the schools as aesthetically modern and beautiful. Across CDSS communities, stakeholders reported that the classroom blocks were well-ventilated, well-lit, and with little disruption from outside noise. Students and teachers reported that overall, the latrines were private and well-functioning, although some students reported having to still fetch their own water from the borehole. In general, stakeholders reported that the number of chairs, desks, and blackboards met classroom needs. WASH FACILITIES FGDs with girls suggested that the construction of the latrines and water availability had a bearing on whether girls chose to attend and stay in school. The toilets ensure accessibility to all learners, provide privacy, and promoted adequate hygienic conditions, especially during girls’ menstrual cycles. Baskets in the latrines allow for disposal of sanitary napkins and 100 percent of girls interviewed said they used the toilets. OVERALL CHALLENGES FACILITIES While stakeholders were pleased with the modern design of the classroom blocks overall, some mentioned that the facilities fell short of expectations. Specifically, stakeholders wishing the schools featured libraries to support student learning was a prominent theme to emerge from the discussions. Additionally, in Kayuni and Njerenje, teachers and students were dissatisfied with the laboratories, citing that construction was either incomplete or of poor quality. Teachers and students at these schools also reported that the roof in the classroom blocks frequently leaks when it rains. Teachers were generally more likely to report dissatisfaction with the school facilities, particularly the staff rooms. In Chilanga and Njerenje, teachers reported that the staff room is too small. During individual interviews, we asked teachers and head teachers their level of general satisfaction with the building and school facilities. Figure 3 shows the majority (63.1 percent) of teachers are satisfied or very satisfied with the new CDSS infrastructure, 20.2 percent are neutral, and 16.7 percent are unsatisfied or very unsatisfied. To compare, we also asked teachers and head teachers about the READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 29 satisfaction they had with the building and facilities of the school where they worked before starting at the AMAA CDSS. Around 46 percent were satisfied or very satisfied, 11 percent felt neutral, 40 percent were unsatisfied or very unsatisfied, while three percent had not worked in a school before. Clearly, despite certain criticisms, the opinions of teachers and head teachers about the AMAA CDSSs is more positive than their opinions of other schools where they worked in the past. Figure 3: Teachers’ satisfaction with building and facilities of the AMAA CDSS TEACHER HOUSING In general, while FGD teacher respondents were appreciative of the teacher housing, they were dissatisfied with the size, number (there are only two houses per school), and lack of electricity of CDSS teacher houses. Teachers claimed the houses were too small for families to live in, and since there were only two teacher housing units per CDSS, their utility was limited. Figure 4 shows the distribution of opinions about the school housing. Twenty-seven percent of all teachers and headteachers live in housing provided by the school. Proportional to their number, there are more headteachers living in housing provided by the school (63.6 percent) than teachers (21.9 percent). During individual interviews, we asked those living in school housing about their satisfaction with the accommodations. In contrast to the school facilities, slightly more than half the teachers and head teachers report being unsatisfied or very unsatisfied (52 percent) with the housing, 22 percent are neutral, and 26 percent are satisfied or very satisfied. Most teachers (75 percent) believe they can find and afford better housing on their own. 16.7% 46.4% 20.2% 14.3% 2.4% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Very satisfied Satisfied Neutral Unsatisfied Very unsatisfied Percentage 30 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Figure 4: Teachers’ satisfaction with the school housing Slightly less than half the teachers and head teachers live in the community where the CDSS is located. Teachers who do not live in the community commute to the school from neighboring villages but also from further away trading posts and towns. During the rainy season, they described how mud can block roads, and teachers oftentimes arrive late to school, ultimately causing students to miss lessons. Teachers explained that if more houses were constructed for teachers, then students’ classes would not be disrupted. SAFETY Perceptions of the safety of the new CDSS facilities were mixed. In some communities there was agreement among teachers, caregivers, and girls that the CDSS is secure, but in other communities, stakeholders reported that they believe the schools are not safe learning environments, because there are no fences to prevent strangers from wandering onto campus. Unclear boundaries and lack of fences have led to disturbances in student learning: girls in several CDSSs pointed to instances when motorcyclists, passing strangers, and community members entered campus and made noise, distracting students from class. Another theme that emerged in a few FGDs was people had broken into the CDSS buildings and stolen belongings from teachers and students. All CDSS communities expressed appreciation for the security guards that work on-site and protect the schools from theft. As mentioned before, despite some stakeholders’ concerns, a comparison of AMAA CDSSs and other secondary schools suggests that learners feel safer and there is more parental and community involvement in AMMA schools (Table 17) BENEFICIARIES’ RECOMMENDATIONS TO IMPROVE SATISFACTION WITH CDSS 9% 17% 22% 30% 22% 0% 5% 10% 15% 20% 25% 30% 35% Very satisfied Satisfied Neutral Unsatisfied Very unsatisfied Percent READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 31 The lack of electricity was framed as both an issue safety and a limitation to education. Stakeholders described how installing electricity in the CDSS would open up additional educational opportunities, like conducting lab experiments or watching English videos. Parents expressed concerns for student safety if they ever need to stay at school late to study without electric lighting. Teachers in treatment communities expressed that they are unsatisfied with the lack of electricity in teacher housing. Some teachers suggested installing solar energy to resolve the issue. Respondents suggested that boarding facilities would also promote girls’ safety. Parents and community leaders advocated for girls boarding instead of commuting to school so they could avoid confronting dangerous strangers on the road. It is important to keep in mind that the AMAA CDSSs are very close to the community primary school. Thus, the distances between students’ homes and the new secondary schools are not meaningfully different from the distances they travelled to reach primary school. Finally, stakeholders mentioned the need for fences with gates to enclose the school property and protect the safety of the students. Students said that a fence would prevent outsiders from entering the school grounds on foot or on bicycle and harassing girls. Looking beyond the construction of the CDSS, stakeholders mentioned that as a next step, the Ministry should provide teaching and learning materials, such as books and lab equipment. Teachers and girls explained that these materials would allow students to engage with the lesson content more actively. CONCLUSIONS School Enrollment and Dropping Out. The newly built AMAA CDSSs have increased Malawi’s capacity to educate secondary school students, particularly in the rural areas of Machinga and Balaka, where there were no public secondary schools before. In the treatment communities, almost 70 percent of the secondary school students are enrolled in the local AMAA CDSS. The benefits of these new CDSSs have reached the comparison communities as well; more than 20 percent of the secondary school students in comparison communities are attending also AMAA CDSS. We observed a slightly lower -2.2 percentage points- reduction in dropouts in the treatment communities; however, the difference is small and not statistically significant. The proportion of students finishing primary school or attending secondary education are also very similar between community groups. However, we do see significant differences in other important aspects. While 7.6 percent of students in treatment communities attend private schools, the percentage is 18.5 among students in comparison communities. The difference in the number of students that had to move away from home to attend secondary schools is also large and significant. While almost 29 percent of the students in treatment communities moved, the percentage of students in comparison communities was significantly higher, reaching over 46 percent. Barriers to Stay in School. The AMAA schools offer access to secondary school to rural students, however dropout rates in the communities we studied continue to be high (38.9 percent of the students in our sample dropped out of school between 2018 and 2021). Dropping out of school is more common among girls than boys. In our sample, around 35 percent of boys left school during the period under study, compared to over 42 percent of girls. Overage students are also more likely to drop out of school than others. Low socioeconomic status, measured as household assets ownership and the education of the mother, is associated with higher risk of dropping out. 32 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Poverty is the main barrier that prevents students from continuing their education. Lack of resources to pay for school fees and other education costs is the most mentioned reason (over 60 percent) for dropping out of school. AMAA schools have contributed to reducing the problem by offering bursaries (30 percent of students in AMAA schools receive financial support to cover fees vs. 14 percent in other schools) and by reducing the need to move to a different area to attend school, which increases education costs. The second most common reason for leaving school was pregnancy. More than a quarter of the girl dropouts reported pregnancy as the reason for abandoning their education. If we also include marriage and having/nursing a baby, this increases to a third of girl dropouts. A common pipeline to dropout that stakeholders described involved girls entering sexual relationships, getting pregnant, sometimes getting married, and then dropping out of school. However, being overage in school is very common and an important contributing factor to the pregnancies problem. Among girls who dropped out before reaching secondary school and have ever been pregnant, around 65 percent had their first pregnancy at age 18 or older. Attitudes and Expectations The attitudes and expectations of girls, boys, parents, and the community in general regarding education, work, marriage, and children were extremely optimistic at baseline. An overwhelming majority of students aspired and expected to attend university, work in their prefer job by age 25, and only get married after 28 years of age. Although aspirations and expectations were slightly higher among those students that remain enrolled in school than among dropouts, they were high for everyone and dropping out does not seem related to lack of aspirations or low expectations, at least as measured. Comparing aspirations and expectations between baseline and endline, we observe that an even larger fraction of students at endline said they wanted to attend college/university, and there are no differences between treatment and comparison communities. However, the expectations were lower from baseline to endline, as fewer students thought they would actually be able to reach their desired education levels. The drop in expectations is more pronounced among those already in secondary school than among students still in primary school, suggesting students’ expectations adjust to reality as they age and move through the education system. Students in treatment communities do not show higher expectations than their peers in comparison communities. Violence Perceptions and Experiences. The students report higher perception of safety and more parental and community involvement in AMMA schools. The results suggest AMAA CDSSs are doing a slightly better job than other secondary schools offering a positive school climate. We asked secondary school students and dropouts about witnessing violence by adults in the community and by a caregiver -against spouse or partner or against siblings. In general, the reports indicate violence is common. A higher percentage of dropouts reported witnessing violence by adults in their community than secondary school students. However, witnessing caregiver physical violence at home is almost 8 percentage points more common among children enrolled in secondary school at endline. The prevalence of secondary school students and dropouts being victims of emotional and physical violence are very similar, around 31 and 54 percent, respectively. However, dropouts reported having been victims of sexual violence in higher proportions than those enrolled in secondary school (33.5 vs 28.3 percent). READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 33 AMAA Teachers. Schools have been able to attract enough teachers to replace the ones that left. In fact, Machinga has been increasing the number of teachers per school since 2019. However, the quality of incoming teachers seems to be deteriorating to some extent, as teachers in cohort 3 have less education and a somewhat less experience than cohort 1 teachers. Teacher retention is relatively high but declining, and for teachers that stayed at least one year since 2019, the retention between 2020 and 2021 is comparable to previous studies for Malawi. In terms of teacher quality, teachers that leave school have more education than teachers that stay, which suggests that the former have better outside options than the latter. Finally, teachers that leave the AMAA schools are more likely to live far from school, which highlights the importance of housing options close to schools as a determinant of teacher retention. Satisfaction with AMAA CDSSs Facilities. Students, their caregivers, teachers and head teachers, and community leaders expressed their satisfaction with the new CDSS facilities. There is agreement that students have benefitted from the construction of new CDSSs closer to their homes. Stakeholders suggested that when it is easy for a student to physically travel to school, there is no need for additional transportation and boarding costs, which are important obstacles to secondary school enrollment and attendance. Respondents expressed appreciation for the CDSS appearance and construction. They find the AMAA schools modern, beautiful, well-ventilated, well-lit, and properly furnished. Separate latrines for teachers and students, and for males and females, provide privacy and promoted adequate hygienic conditions, especially during girls’ menstrual cycles. This is a highly appreciated feature the AMMA schools. However, stakeholders were disappointed by a lack of several amenities in the CDSS, such as electricity, fences to clearly demarcate school boundaries, a library, and instructional materials such as textbooks and lab equipment. A main concern among teachers relates to the insufficient supply of housing units, as well as housing units that are smaller than needed, and lack electricity and water. Given the lack of quality housing alternatives around these rural communities, this affects teachers’ satisfaction, and possibly retention. RECOMMENDATIONS Having enough schools close to the rural population increases access to secondary education and reduces inequalities. School construction efforts should be continued in other areas. Having schools is a necessary condition to increase education, but it is not sufficient. Poverty limits the ability of families to keep their children in school. Although in 1994 Malawi introduced Free Primary Education, schools collect some fees from primary students; secondary schools, such as AMAA CDSSs, charge around MKW 10,000 per term; and, in addition, families need to cover for other costs, such as uniforms or exams fees. AMAA provided bursaries to some secondary students which helped but more support is needed to keep girls and boys enrolled in school. In addition to poverty, girls face an additional challenge to stay in school: pregnancies. Policies that promote age-appropriate sexual and reproductive health education for girls and boys should be a priority. This would not only affect school enrollment, but also reduce the transmission of HIV and other STIs. At the same time, stakeholders should recognize that the problem of pregnancies is tied to 34 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA the high percentage of students who are overage. Policies should aim to ensure children start school on time and reduce grade repetition rates, thereby reducing the high prevalence of students who are well beyond primary school age but still in Standards 5-8. For example, 22 percent of students in Standard 8 at baseline were age 18 or older, when the expected age for Standard 8 is 12. This means that policies aimed at improving secondary school transition rates would be remiss if they do not begin by addressing the issues with primary schools that are causing the vast majority of students to be well above the expected age for their grade. Finally, attention is needed to improve teachers’ housing options. Finding teachers for the new schools is the responsibility of the MoEST and is not easy. Many teachers do not want to live in rural areas far from the trade centers, health facilities, and recreational facilities that are available in urban areas. The construction of only two teacher houses at each school, and the limited amenities they provide, appears insufficient to attract and retain quality teachers. Increasing the number and size of the houses and adding features to increase comfort, could be a strategy for attract qualified teachers and reducing turnover. Alternatively, monetary, and non-monetary incentives could be explored, though the evidence suggests housing near the school is essential. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 35 REFERENCES Adams, B., Wasikhongo J., and Nahemow N. (2014) Socio-economic Status and Educational Aspirations in Uganda, Canadian Journal of African Studies, pp. 221-230. Baird, S. C. McIntosh, and B. Ozler (2011) “Cash or Condition? Evidence from a Cash Transfer Experiment” The Quarterly Journal of Economics (2011) 126, 1709–1753. Beutel, Ann, and Anderson, K. (2007) "The educational expectations of South African youth." Sociological Focus, 40(4), pp. 348-369. Burde, D., and Linden, L. (2013) Bringing Education to Afghan Girls: A Randomized Controlled Trial of Village-Based Schools." American Economic Journal: Applied Economics, 5 (3): 27-40. Chowa, Gina, and William Elliott III. (2007), Increasing Parental Educational Expectations for Children in Sub-Saharan Africa: The Potential Role of Assets, Centre for Social Development, no. 07-18. Duflo, E. (2001) Schooling and labor market consequences of school construction in Indonesia: evidence from an unusual policy experiment. The American Economic Review, 91(4), pp. 795–813. Duflo, E. (2004) The medium run effects of educational expansion: evidence from a large school construction program in Indonesia. Journal of Development Economics, Elsevier, vol. 74(1), pp. 163- 197. Filmer, D. (2007) If you build it, will they come? School availability and school enrolment in 21 poor countries. The Journal of Development Studies, 43:5, pp. 901-928. Guerrero G., Sugimaru C., Cussianovich A., Fraine, B., and Cueto, S. (2016), Education Aspirations among Young People in Peru and their Perceptions of Barriers to Higher Education, Young Lives Working Paper 148 Young Lives, Department of International Development at the University of Oxford. Handa, S. (2002) Raising primary school enrolment in developing countries: The relative importance of supply and demand. Journal of Development Economics, 69(1), pp. 103–128. Hainmuller, J. 2012. “Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies.” Political Analysis 20: 25-46. Kadzamira, E. (2003), Malawi’s Experience in Promoting Girls’ Education, University of Malawi Centre for Educational Research & Training. Kazianga, H., Levy, D., Linden, L., and Sloan, M. (2013) The Effects of "Girl-Friendly" Schools: Evidence from the BRIGHT School Construction Program in Burkina Faso. American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pp. 41-62. Keleher, H., Parker, R., Abdulwadud, O. and Francis, K. (2009), Systematic review of the effectiveness of primary care nursing. International Journal of Nursing Practice, 15: 16-24. doi:10.1111/j.1440- 172X.2008.01726.x 36 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Kendall, N., 2006. Strengthening gender and education programming in the 21st century. In: Equip 2 Working Paper, USAID, Washington, DC. Lavy, V. (1996) School supply constraints and children’s educational outcomes in rural Ghana. Journal of Development Economics, 51(2), pp. 291–314. McConnell, C. and Mupuwaliywa, M. (2016), Keeping Girls in School: Situation Analysis for Malawi, World Bank Group. Menendez, A., G. Haugan, and Z. Grotophorst (2018) Apatseni Mwayi Atsikana Aphunzire AMAA, Evaluation and Research. Baseline Report for School Construction Activity in Machinga and Balaka Districts. NORC at The University of Chicago – USAID https://pdf.usaid.gov/pdf_docs/PA00TK77.pdf Menendez, A, G, Haugan, M. Rezwan, N. Taha, and P. Pepitone (2022) Barriers to Girls’ Education in rural Malawi. Evidence from Machinga and Balaka districts. Miguel, E. and Kremer, M. (2004) Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities. Econometrica 72 (1), pp. 159-172 Mulkeen, Aidan (2010). Teachers in Anglophone Africa: Issues in Teacher Supply, Training, and Management. Development Practice in Education. Washington, DC: World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/13545 License: CC BY 3.0 IGO. Omoeva, C. and R. Hatch, with B. Sylla (2014) Teenage, Married and Out of School. Effects of early marriage and childbirth on school dropout. FHI 360. Tafere, Y., (2014) “Education Aspirations and Barriers to Achievement for Young People in Ethiopia, Young Lives” Working Paper 120, Young Lives, Department of International Development at the University of Oxford. World Bank, (2010), The education system in Malawi, World Bank Working Paper 182, Africa education country status report, World Bank. Younger, S. (2000) The incidence of public services and subsidies in Peru: evidence from household surveys. Cornell Food and Nutrition Policy Program working paper, No. 103, Cornell University, Ithaca, NY, http://papers.ssrn.com/sol3/papers.cfm?abstract_id/14436602 READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 37 ANNEX I: EVALUATION STATEMENT OF WORK DRAFT 1 STATEMENT OF WORK FOR The Apatseni Mwayi Atsikana Aphunzire (AMAA) Evaluation, Malawi February 22, 2018 PURPOSE OF THE EVALUATION: OBJECTIVES, AUDIENCES AND USES Apatseni Mwayi Atsikana Aphunzire (AMAA) is supported by USAID’s Let Girls Learn Challenge Funds awarded by the Office of Gender Equality and Women’s Empowerment (E3/GenDev) to Save the Children, Inc. in December 2016. AMAA is implemented by Save the Children Inc. in collaboration with several partners through Cooperative Agreement AID-OAA-A-17-00001. AMAA is currently managed by the E3/Office of Gender Equality and Women’s Empowerment. Plans are underway to transfer AMAA management to USAID/Malawi in FY2018. The evaluation of AMAA’s school construction effects is funded by E3/GenDev. The AMAA evaluation operations will be carried out by NORC, through the Reading and Access Evaluations contract (Time and Material): GS-10F-0033M/AID-OAA-M-13-00010. The Reading and Access Evaluations contract is managed by the E3/Office of Education (E3/ED). A GenDev team member will serve as the Activity Manager of the AMAA evaluation operations. The audiences for this evaluation comprise USAID Operating Units (OU), notably, USAID/Malawi, the Africa Bureau, E3/GenDev, the E3/Education Office; and the E3/ Energy and Infrastructure Office. Other important audiences are the Government of Malawi, primarily, Malawi’s Ministry of Education, Science and Technology (MoEST), and donors committed to building and/or supporting schools, such as, Japan International Cooperation Agency (JICA), Department for International Development (DFID), the World Bank and the World Food Program. Save the Children and its implementing partners are key audiences too. AMAA’s Whole-of-Girl programming package is formed of multiple district specific interventions in five districts. However, this AMAA evaluation focuses only on the set of secondary school construction interventions implemented in two districts, Machinga and Balaka. The objectives of the AMAA evaluation are to: • Review existing evidence on the relationship between new school construction and enrollment as relevant to Malawi and interpret the findings of the AMAA evaluation in the light of this evidence; • Building on the overall AMAA logical framework and theory of change, prepare a theory of change for AMAA new secondary school construction in collaboration with implementing partners and other stakeholders including USAID/Malawi; • Evaluate and report on the effects of the AMAA new secondary school construction relating to completion of primary education by girls and boys, transition of girls and boys from upper primary to the new AMAA secondary schools, and attendance and retention of girls and boys in 38 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA the new AMAA secondary schools. In addition, the evaluation will examine the effects of AMAA new secondary school construction on perceptions of safety by girls and boys. The evaluation will also assess whether girls, boys, teachers, parents, and community members are satisfied with the school construction design and facilities. • Assess the AMAA approach to constructing new secondary schools. The findings, conclusions and recommendations of this evaluation can be applied to future school construction in Malawi and other places. The data generated by this evaluation will contribute toward building the growing body of evidence on the relationship between school construction and enrollment, informing new school construction programming for adolescent girls. SUMMARY INFORMATION Strategy, Project, or Activity Name Apatseni Mwayi Atsikana Aphunzire (AMAA) in Malawi Implementer Save the Children, Inc. Cooperative Agreement/Contract # Cooperative Agreement NoAID-OAA-A-17-00003 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $ 10.5 million Life of Strategy, Project, or Activity December 15, 2016 to December 14, 2021 Active Geographic Regions AMAA will be implemented in five districts in Malawi: Balaka, Machinga, Phalombe, Chikwawa and Mzimba districts. Ten new schools will be built in the Machinga and Balaka districts. Development Objective(s) (DOs) E3/Office of Gender Equality and Women’s Empowerment PAD DO2: Gender-informed approaches scaled in USAID programs and processes. USAID/Malawi, CDCS DO1: Social Development Improved. USAID Office E3/Office of Gender Equality and Women’s Empowerment in partnership with USAID/Malawi BACKGROUND 3A. DESCRIPTION OF THE PROBLEM, DEVELOPMENT HYPOTHESIS, AND THEORY OF CHANGE 3A.1. THE PROBLEM: Since Malawi introduced free primary education in 1994, the numbers of boys and girls enrolling in the first four years of primary school has grown exponentially. Commendable progress has been made towards gender parity at the lower primary level, however, from Standard 5, the number of girls dropping out far surpasses that of boys. For example, while the survival rates to Standard 5 have remained similar for boys and girls (58% and 59% respectively in 2012; 66% and 62% in 2013), the survival rates to Standard 8 have consistently been higher for boys than for girls (41% and 35% respectively in 2012; 35% and 27% in 2013). Girls also tend to fare worse than boys in examinations. As READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 39 an example, in 2014 73% of boys who sat the Primary School Leaving Certificate Examination (PSLCE) passed compared with 61% for girls13. Only a small percentage of girls and boys who pass are able to go to secondary school due to lack of schools and trained teachers. Although the government tries to maintain an equal selection of girls and boys for secondary school, fewer girls than boys’ complete secondary school due to long distances, lack of school fees, family responsibilities, marriage, and pregnancy. In 2015, there were 46,829 boys in Form 4, the last year of secondary school, versus just 38,654 girls14. Many girls are leaving the education system without having attained basic literacy, numeracy, or essential life skills, making them more dependent on marriage and a husband’s support. Societal norms that lead to gender inequalities make girls responsible for the majority of household chores, resulting in them arriving late to school, or reducing time for homework, causing them to fall behind. This gender bias and society’s perceptions of what a girl’s role can and should be and what she is capable of, also influences whether a guardian will allocate scarce resources to support a girl’s education. Girls are also more likely to be victims of violence at school, or on the way to and from school. Long distances can prevent both boys and girls from attending school. However, this affects girls differently as the farther a girl must travel, the greater the concerns for her safety. Parents may be reluctant to allow their daughters to travel long distances, or the girl herself might think school is not worth the additional risk. At school, sexual assault and rape by teachers and classmates is commonplace15. In the Government’s Violence Against Children Survey, 2013, 22.8% of girls (age 13-17) said they had experienced sexual violence in the previous 12 months. Yet victims of physical and sexual violence rarely seek help. Only 14% of girls aged 13-17 said they had sought help after a sexual or physical attack16. Sexual violence and abuse, the lack of sexual and reproductive health knowledge and youth friendly services, and girls’ inability to negotiate safe sex due to restrictive gender norms can mean many girls are forced to drop out of school due to pregnancy. Nearly 29% of girls under 19 have started childbearing, an increase of 4% on the previous year17, and a trend that heavily impacts access to upper primary and secondary school, because, as soon as a girl is pregnant she must drop out of school for at least a year18. Additionally, a portion of these girls are married or soon become married; 46% of girls are married before they reach the age of 1919. Girls who do return to school are often quite far behind, and older than peers in their class making it likely they will drop out again. In fact, 75% of all new HIV infections are among adolescent girls and young women20 Due to the prevailing cultural and societal norms21, young girls are the least able to negotiate safe sex, or resist unwanted sexual advances from men, including their teachers. Unprotected sex also puts adolescent girls at risk of contracting HIV. 3A.2. AMAA’S THEORY OF CHANGE: To address these multiple interlinked barriers that limit a girl’s ability to develop, Save the Children Federation, Inc. (SC), together with its partners Grassroot Soccer (GRS), and Forum for African 13 Education Management Information System (EMIS) Malawi, 2014 14 EMIS, 2015 15 Malawi Gender Assessment: Gender Based Violence and Girls’ education Outcomes, 2016, US State Department 16 Violence Against Children and Youth survey, 2013 17 Malawi Demographic and Health Survey, 2015-2016 18 Violence Against Children (VAC) Survey, Government of Malawi, 2014 19 UNICEF, State of the World’s Children, 2016 20 DHS 2015 21 Malawi ranks 145 out of 149 countries on the UN’s Gender Inequality Index, according to the HDI 2016 40 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Women Educationalists – Malawi Chapter (FAWEMA), are implementing Apatseni Mwayi Atsikana Aphunzire (AMAA), which translates in Chichewa to Give Girls a Chance to Learn. The project adopts a “whole of girl approach” in its design and interventions. AMAA’s Whole of Girl conceptual approach (Figure 5) recognizes that girls are actors in their own personal development, but contends girls’ development cannot be considered outside a social web of relationships, influential actors, norms, and systems: The AMAA theory of change aims at addressing the five main common barriers to girls’ education; school fees, early pregnancy, early marriage, distance to secondary school and lack of space in secondary school. AMAA intends to address these common barriers by delivering three outcomes for improving the girl’s agency and self-efficacy, improving access to education and performance for girls and improving the community support for girls’ education and safety. This will be at the girl as an individual, the family, the community, school, and the system level to ensure context specific interventions that fully addresses the different structural problems facing girl’s education in Malawi. Figure 5. AMAA Theory of Change Diagram: The Whole of Girl Approach 3B. SUMMARY STRATEGY/PROJECT/ACTIVITY/INTERVENTION TO BE EVALUATED 3B.1. THE LET GIRLS LEARN WHOLE-OF-GIRL APPROACH: In March 2015, the U.S. Government launched Let Girls Learn (LGL), a global initiative which builds upon existing efforts and invests in new programs to expand education for adolescent girls. LGL seeks to ensure that adolescent girls around the world can access quality education enabling them to reach their full potential. LGL activities adopt a Whole-of-Girl approach by recognizing that multiple solutions are needed to address the complex challenges faced by girls. Barriers to girls’ education pose formidable challenges. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 41 The Whole-of-Girl approach is based on the recognition that discriminatory gender norms motivate some parents to invest in educating their sons rather than their daughters. Girls attending school risk harassment and violence on their way to school and in school. Girls drop out of school owing to pregnancy and early marriage. In some contexts, parents and teachers tend to support boys rather than girls to make headway in their studies and careers. Whole-of-Girl interventions foster an enabling environment for adolescent girls’ education, safety, and well-being, wherein girls increase self-efficacy and are valued by their families, communities, and other institutions. 3B.2. AMAA: AMAA is delivering a range of district specific activities in Malawi to mitigate the barriers to girls’ enrollment and retention in school. The project works in five priority districts targeting over 60,000 girls aged 10-19 in both upper primary and secondary schools. Led by Save the Children, AMAA works with local and international NGOs, as well as Peace Corps, the World Food Program and the Ministry of Education, Science and Technology (MoEST) in Malawi to mitigate the barriers preventing girls’ enrollment and retention in school. AMAA developed a package that includes targeted yet competitive bursaries, construction of secondary schools, extracurricular and in school activities to empower girls and strategies to transform gender norms and reduce school-related gender based violence (SRGBV). The AMAA project focuses on the reduction of early and child marriage and teen pregnancy for the purpose of increased school access, performance, and retention. For girls to achieve academic success, they must enter school, stay in school, be learning while in school, and be healthy and always supported by their community. AMAA implements a coordinated set of interventions with the following activities: ● increasing access to schools and safety for girls by constructing of one 50-bed girls’ dormitory and ten new secondary schools; protecting girls from gender-based violence by improving reporting and referral systems; and involving fathers from the community in AMAA “Real Fathers” program on building healthy father-daughter relationships; ● offering additional learning opportunities and support for all adolescent girls to continue their education by establishing school buddies for girls who are caring for babies; and training male and female teachers to inspire and support girls to remain in school; and ● ensuring that girls are receiving a practical, high quality education by improving the academic and teaching performance of teachers in English, science, and math; and inviting girls to yearly science and math camps. EVALUATION QUESTIONS The evaluation questions broadly fall into two categories. The Part 1 evaluation questions focus on the effects of AMAA new secondary school construction on girls, boys, parents, teachers, and communities. It is essential for the evaluation team to define key terms in consultation with USAID and Save the Children. These key terms include school “retention,” “transition,” “completion, “drop-out,” “repetition,” and “held-back” etc. The evaluation team should also specify the physical and socio￾emotional aspects of safety (Q.5) and the attributes of school design (Q.7) that will be assessed by beneficiaries. The Part 2 evaluation question focuses on assessing the AMAA approach to new school 42 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA construction. The areas of inquiry for this question should be determined in consultation with USAID and Save the Children. PART 1: These evaluation questions22 will capture the effects of AMAA’s new secondary school construction interventions on beneficiary girls, teachers, parents, and communities. Evaluation.Q.1: What effect does AMAA new secondary school construction have on the attitudes and expectations of girls, boys, parents, and communities in the areas served by the new AMAA constructed new schools regarding: ● Primary school retention and completion and transition to secondary schools? ● Interest in continuing girls’ (boys’) education and/or vocational training; ● Marriage and pregnancy; ● Future work/employment/career; and ● Other? Evaluation.Q.2: What effect does AMAA new secondary school construction have on primary school completion and transition; and secondary school enrollment, attendance, promotion and retention of girls and boys? Evaluation.Q.3: How are the new AMAA constructed secondary schools helping girls to overcome identified barriers, such as: education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school and lack of space in secondary schools? Are there additional barriers to accessing schools faced by girls? Evaluation.Q4: What are the reasons for girls and boys to: (a) drop out of school; (b) repeat, and (c) be held back? Evaluation.Q.5: What effect does AMAA new secondary school construction have on the perceptions of girls and boys regarding physical and socio-emotional aspects of safety? Evaluation.Q.6: What effect does AMAA new secondary school construction have on attracting and retaining teachers? Evaluation.Q.7: Are girls, their parents/guardians, and community members satisfied with the construction design and facilities of the new schools? The construction design attributes should be further specified in collaboration with USAID. PART 2: This evaluation question will assess the AMAA approach to constructing schools in Malawi Evaluation.Q.8: Assess the approach adopted by AMAA to construct schools in Malawi. How were challenges overcome? What are the strengths and limitations of the school construction experience? What are the promising practices and lessons for USAID and its partners? 22 The AMAA evaluation questions are focused on the AMAA new secondary school construction package. This package supports AMAA IR 1.1 Improved secondary school enrollment. In 2018-19, AMAA is building 10 new secondary schools, five in the Machinga district and 5 in the Balaka district. The school buildings will be constructed by an engineering company. The community will be involved in building the latrines and the teachers’ quarters. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 43 EVALUATION DESIGN AND METHODOLOGY The evaluation team is tasked with designing and implementing an evaluation of AMAA new secondary school construction in Malawi’s Machinga and Balaka districts. Overall, this evaluation design should be developed in consultation with USAID and the AMAA team. To answer the Part 1 evaluation questions, the AMAA evaluation team will adopt a mixed methods approach, collecting qualitative and quantitative data through literature review and document analysis, surveys, key informant interviews, and focus groups. Malawi’s Education Management Information System (EMIS) is an important source of documentary information on school enrollment and attendance. The evaluation design should leverage EMIS and attendance information maintained by schools if and when appropriate and feasible. The evaluation team could apply a quasi-experimental design using treatment and comparison groups to answer certain Part 1 evaluation questions. If a quasi-experimental design is used, the treatment group(s) would be drawn from the treatment areas, or the areas adjacent to the new schools. The treatment groups should comprise girls and boys eligible to attend the new secondary schools and/or those who have transitioned successfully to the new secondary schools. The control or comparison groups would be drawn from areas that are similar to the treatment areas on key attributes, but currently do not have secondary schools close to primary schools. The analysis and presentation of data gathered in response to the Part 1 evaluation questions should be informed by the literature and available evidence on school enrollment and retention if the studies are relevant to Malawi. To answer the Part 2 evaluation question, the evaluation team will analyze relevant documents, such as, AMAA school construction proposal(s), work-plan(s) and budgets, Initial Environmental Examination (IEE) and site specific Environmental Monitoring and Mitigation Plans (EMMP), the AMAA progress reports to USAID, documentation relating to procurement, school construction designs (drawings and specifications) and contractor/community plans. To answer the Part 2 question, the evaluation team will also gather information from USAID staff and non-USAID stakeholders. The USAID stakeholders include the Malawi Mission, the Africa Bureau, E3/GenDev, E3/Education, E3/Environment offices, E3/Energy and Infrastructure, E3/Water. The non￾USAID stakeholders include Save the Children and AMAA staff, key personnel from Malawi’s Ministry for Education, Science and Technology (MoEST) and other Malawi government agencies supporting school construction; and community members. In alignment with the requirements and standards in Section C: Scope of Work of the Reading and Access contract, the evaluation team will undertake and complete the following tasks, including: • Finalize the Part 1 and Part 2 evaluation questions drafted to-date in consultation with USAID and AMAA team; • Prepare the Evaluation Design Report and the corresponding evaluation work-plan and detailed evaluation budget using the templates recommended by the E3/Education R&A Contracting Officer’s Representative (COR). The Evaluation Design Report should: o Describe the evaluation methodology to collect and analyze quantitative and qualitative data, and disaggregate data by sex, age, and other relevant variables; 44 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA o Provide information on the sampling frame, power calculations, control, and treatment groups; o Discuss internal and external validity and how generalizable the results will be; o Identify the strengths and limitations of the proposed methodology as well as evaluation implementation challenges and proposed solutions. o Discuss the approach to optimizing cost-efficiency; o Describe the plan to leverage local knowledge to benefit the evaluation team; o Describe the data collection tools for the surveys, focus group discussions and key informant interviews; o Use the evaluation report template(s) recommended by the R&A COR. • Prepare the data collection tools; • Recruit and deploy an evaluation implementation team with the recommended qualifications, including Malawian nationals; • Implement the AMAA evaluation in close consultation with USAID and the AMAA team; • Report the findings, conclusions, and findings of the AMAA evaluation in the context of the needs of adolescent girls in Malawi and the AMAA school construction experience; and • Implement the activities to learn from the AMAA evaluation’s findings, conclusions and recommendations during the baseline, midline and end-line stages. DELIVERABLES AND REPORTING REQUIREMENTS 6A. Deliverables With reference to the requirements for deliverables in the Reading and Access Evaluations contract, the evaluation team will deliver the following products: • In 2018 by March 31st, an Evaluation Design Report with a work-plan and detailed budget for the AMAA evaluation with an in-house presentation on the evaluation design to USAID and the AMAA team using audio-visual media to connect audiences; • The set of data collection tools; • Briefings for USAID/Malawi and AMAA team during data collection visits in Tanzania; • In 2019 by June 30th, a draft Part 1 baseline report, the final baseline report, and a presentation to USAID and the AMAA team, using audio-visual media to connect audiences; • In 2019 by September 30th, a draft Part 2 evaluation report, the final Part 2 evaluation report and presentation to USAID and the AMAA team, using audio-visual media to connect audiences; • In 2021 by July 30th, a draft Part 1 end-line report and the final Part 1 end-line evaluation report with an in-house presentation to USAID and the AMAA team, using audio-visual media to connect audiences; • In 2021 by August 30th, a 1-2 page summary Briefing Note READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 45 • In 2021 by September 15th, hold a webinar on the evaluation’s findings, conclusions and recommendation for Agency-wide stakeholder audiences and post the webinar recording on Agency web sites recommended by USAID. 6B. Reporting The evaluation reports (Part 1 baseline, Part 2 final report, and Part 1 end-line) will discuss the AMAA evaluation’s findings, conclusions, and recommendations in the context of the needs of girls in Malawi and the AMAA school construction experience. Each evaluation report (Part 1 baseline, Part 2 final report, and Part 1 end-line) should contain the following components: ● Executive Summary; ● Introduction (including evaluation purpose, audience, anticipated uses, and questions); ● Program Background (overview of the needs of adolescent girls in Malawi, the AMAA Project and the AMAA new school construction activities); ● Methodology (overview of data collection and analysis methods, including limitations, anticipated challenges, and solutions); ● Findings, conclusions, and recommendations for each evaluation question in the context of relevant needs of adolescent girls in Malawi and the AMAA school construction experience implementation experience; and ● Annexes (including evaluation SOW, references, list of respondents, and data collection instruments). The AMAA evaluation reports will meet the evaluation quality criteria described in the USAID Evaluation Policy (see text box below): USAID EVALUATION POLICY, APPENDIX 1 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 46 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA EVALUATION TEAM COMPOSITION Within the Labor Categories described in the Reading and Access Evaluations contract, NORC will select an evaluation team that includes a Principal Investigator, statistician(s), gender in education specialist(s), architect/engineer, administrative associate(s), and in-country data collectors. In alignment with the requirements and standards in Section C: Scope of Work of the Reading and Access Evaluations contract, the AMAA evaluation team is tasked with designing and implementing the evaluation to answer the Part 1 and Part 2 in consultation with USAID and the AMAA team. The WW evaluation team will collectively reflect the following qualifications scaled appropriately to senior, mid-level, and junior levels: • Graduate degree(s) in the social sciences; • Expertise in applying a variety of methods to gather and analyze quantitative and qualitative data, as well as designing and implementing impact evaluations and performance evaluations; • Knowledge of USAID’s performance monitoring and evaluation guidance; • International development programming experience; • Demonstrated experience in gender equality and women’s empowerment and/or programming; • Specialist knowledge on the needs of adolescent girls and education systems in Malawi; • Professional architect/engineer qualifications with knowledge of USAID’s construction standards and extensive school construction experience; • Experience in managing projects/activities funded by USAID; • Specialist expertise in working with databases, and software to collect and analyze data and to disseminate findings; • Data collection, data analysis, report writing and communication skills, including languages spoken in the WW implementation areas; and • Excellent ability to work collaboratively with evaluation team members, USAID, WEI/B, and its partners. The Evaluation Design Report will propose the staffing plan for this evaluation, including, specific positions and CVs for proposed individuals to serve in the evaluation team positions. Each evaluation team member will sign USAID’s conflict of interest statement before conducting any documentary or field research. To leverage local knowledge, the contractor is encouraged to use host-country evaluation specialists with the requisite qualifications. EVALUATION SCHEDULE READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 47 Tasks and Deliverables Timeline Illustrative Dates 1. Submit draft Part 1 Evaluation Design Report to Activity Manager (AM) 10 business days following receipt of USAID approval of the evaluation Statement of Work March 23, 2018 2. Presentation on Part 1 Evaluation Design Report to USAID, and WEI/B Within 10 business days following AM’s feedback on the draft Evaluation Design Report April 2, 2018 3. Submit Final Part 1 Evaluation Design Report, incorporating USAID’s feedback on the draft 10 business days after receiving written feedback from all USAID stakeholders on the draft Evaluation Design Report April 15, 2018 4. Submit draft Part 1 data collection tools for review and incorporate stakeholder feedback March 2018 April 2, 2018 5. Brief USAID/Tanzania and WEI/B, collect baseline data and draft baseline report April-May, 2018 6. Submit Draft Baseline Report to AM 40 business days following completion of data collection activities 7. Present baseline evaluation findings and recommendations to USAID audiences and WEI/B stakeholders Within 10 business days after submission of Draft Evaluation Report 8. Submit Final Baseline Report to AM Within 15 business days following receipt of final feedback from all reviewers 9. Brief USAID/Tanzania, and WEI/B, collect midline data and start drafting midline report November – December 2019 10. Submit Draft Midline Evaluation Report to AM 45 business days following completion of data collection activities 11. Present midline evaluation findings and recommendations to USAID audiences and WEI/B stakeholders Within 10 business days after submission of Draft Evaluation Report 12. Submit Final Midline Report to AM Within 15 business days following receipt of final feedback from all reviewers 13. Brief USAID/Tanzania, and WEI/B, collect end-line data and start drafting end-line report May, 2021 14. Submit Draft End-Line Report to AM 45 business days following completion of data collection activities 15. Present end-line evaluation findings and recommendations to USAID audiences and WEI/B stakeholders Within 10 business days after submission of Draft Evaluation Report 48 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Tasks and Deliverables Timeline Illustrative Dates 16. Submit Final End-Line Report and Briefing Note to AM Within 15 business days following receipt of final feedback from all reviewers 17. Provide evaluation debriefing to AM and E3/Education Office Evidence Team Within 10 business days following the submission of the Final Evaluation Report 18. Hold and record webinar on evaluation Within 5 business days following the submission of the Final Evaluation Report OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 49 ANNEX II: EVALUATION METHODS AND LIMITATIONS SAMPLING STRATEGY NORC took advantage of the site selection process that considered a group of similar communities as candidates for treatment. Treatment was only assigned to a subgroup of communities, allowing NORC to draw comparison groups from the rest of the list. The Treatment communities’ selection process is detailed below, followed by NORC’s process in selecting Comparison communities. The selection of Treatment communities that will receive the AMAA secondary schools was done by a committee made up of USAID/Malawi, Save the Children, and MoEST officials, and was supported by technical teams. USAID/Malawi selected the initial pool of candidate communities using the following criteria: ■ Primary school pass rate ■ Standard eight enrolment ■ Located at least 10km from Community Day Secondary School (CDSS) The selection committee visited these communities to meet with the primary school head teacher, community/school structures and community leaders and conduct a community assessment. The committee then selected communities from this list based on their own preferences. Each selected community was then assessed on technical suitability with respect to the following criteria: ■ Large enough to cater for the size of the structures to be constructed ■ Land voluntarily donated by the community and land user ■ Not owned by the church ■ Soil suitable for construction ■ Nearby water source ■ Environmental impact of the construction Some selected communities were rejected on technical grounds and had to be replaced by others in the original list of candidates. This occurred several times, giving the selection process a high degree of (unintended) randomness and allowing the evaluation team to use the initial list as a source for selecting comparison communities. There are two important points to note: (1) none of these technical construction criteria are likely to have any direct impact on the outcomes of interest in this evaluation and therefore are unlikely to create any bias, and (2) treatment and comparison communities do not need to be identical, as the methodology uses statistical approaches to deal with differences if necessary. A. MACHINGA DISTRICT During the initial selection process the committee considered a total of 14 communities: Chisani, Kalambo, Kayuni, Matanda, Mkotamo, Msasa, Mwalasi, Nansato, Nkasaulo, Ntalala, Nyenyezi, Thobola, Msasa, Ntulira, and Nathendo. 50 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA From this list, the committee initially selected the following communities in Machinga to receive the secondary school construction treatment: Kayuni, Matanda, Mkotamo, Mwalasi, and Ntalala. NORC then selected 4 comparison communities from the initial list of candidates, discarding several communities because their proximity to treatment sites. The four communities selected to be used as comparison were: Nkasaulo, Nansato, Ntulira, and Nathendo. None of them are located within 10km from an existing or planned Community Day Secondary School (CDSS). To create an even number with the treatment groups, NORC needed to add one additional comparison site, so Chisese was selected using the same criteria USAID/Malawi used in creating the initial list of candidate communities (i.e., primary school enrollment, primary certification pass rate, and distance from Community Day Secondary Schools (CDSS)). Table 23 shows each community’s enrollment of boys and girls in standard 8, primary school pass rate in numbers and in percentages, and whether the community is in the treatment or comparison group. Table 23. Treatment and Comparison Communities in Machinga District Communities Considered for Treatment* Name of Community Standard 8 Enrollment Primary School Pass Rate Male Female Total Male Female Total % Group Chisani 44 24 68 39 12 51 75% Matanda 33 30 63 19 12 31 49% Treatment Mwalasi 30 24 54 20 15 35 65% Treatment Msasa 61 40 101 44 33 77 76% Ntulira 32 12 44 16 1 17 39% Comparison Kalambo 41 48 89 27 25 52 58% Mkotamo 41 61 102 24 17 41 40% Treatment Thobola 23 34 57 19 29 48 84% Nansato 15 15 30 13 12 25 83% Comparison Ntalala 19 19 38 17 11 28 74% Treatment Kayuni 39 29 68 25 13 38 56% Treatment Nyenyezi 21 20 41 17 8 25 61% Mkasaulo 37 35 72 30 27 57 79% Comparison Nathendo 22 30 52 16 15 31 60% Comparison Comparison Community Selected Using Matching Chisese 22 24 46 18 13 31 67% Comparison * Considered as potential treatment communities by USAID/Malawi, SC, and Malawi education officials. B. BALAKA DISTRICT The selection of treatment and control communities in the district of Balaka followed the same approach used in Machinga. The original communities under consideration were: Ng'onga, Chilanga, Makanjira, Ngwengwe, Nagulukitiche, Namikombe, Ntalikachao, Muthe, Thundu, Njerenje, Mitengwe, Namichere, and Chikoleka. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 51 From this list, the committee chose Chilanga, Makanjira, Namikombe, Nagulukitiche, Njerenje, and Namichere for treatment. However, on May 9, we were informed that Nagulukitiche could not be considered for treatment as it was closer to Chilanga than desirable. The site was then replaced by Ngwengwe. However, this community was also rejected and then replaced by Muthe. NORC then selected the following comparison communities by the same method used in Machinga: Ng’onga, Ngwengwe, Nagulukitiche, Thundu, Mitengwe, and Chikoleka. On May 28, after the baseline data collection took place, NORC learned that Muthe was no longer deemed feasible for construction and thus the communities of Chawanda, Mzimundilinde, and Njese were under consideration as replacement treatment sites. On June 26, NORC learned that the replacement site would be Mzimundilinde. NORC worked with local partner Invest in Knowledge (IKI) to collect data in this new site and add it to the evaluation. NORC had already collected data in Muthe during the baseline and decided to include Muthe as a comparison community, increasing the number of comparison communities to seven. Table 24 shows each community’s enrollment of boys and girls in standard 8, primary school pass rate in numbers and in percentages, and whether the community is in the treatment or comparison group. Table 24. Treatment and Comparison Communities in Balaka District Communities Considered for Treatment* Name of Community Standard 8 Enrollment Primary School Pass Rate Male Female Total Male Female Total % Group Ng'onga 27 25 52 22 24 46 89% Comparison Chilanga 29 15 44 27 13 40 90% Treatment Makanjira 12 17 29 12 15 27 93% Treatment Ngwengwe 10 14 24 10 14 24 100% Comparison Namikombe 17 17 34 15 17 32 94% Treatment Nagulukitiche 18 22 40 15 18 33 82% Ntalikachao 30 35 65 26 26 52 80% Comparison Muthe 47 32 79 40 25 65 82% Comparison Thundu 17 33 50 10 23 33 66% Comparison Njerenje 21 24 45 18 20 38 84% Treatment Mitengwe 41 39 80 35 25 60 75% Comparison Namichere 24 22 46 24 21 45 97% Treatment Chikoleka 45 43 88 40 41 81 92% Comparison Mzimundilinde 19 20 39 19 20 39 100% Treatment * Considered as potential treatment communities by USAID/Malawi, SC, and Malawi education officials. 52 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Table 25: Evaluation Matrix Question Data Collection Activity Data Type/s Indicators Location and Group Date Analyses Q1. What effect does embedding a new CDSS in a community have on (1) primary school completion and transition; and (2) secondary school enrollment, attendance, promotion and retention of girls and boys in the community? Students - Administrative School Data Students Survey Longitudinal data students; tracking 2018 cohort Std5-Std8 CENSUS Longitudinal data students; tracking 80 students per school Enrollment, promotion, repetition, completion, and transition to secondary school Socio￾demographic and economic background, older sibling information, HHLD contact information Machinga and Balaka Treatment and Comparison May 2018 Mar 2019 Mar 2020 Mar 2021 Regression analysis with controls and/or propensity score matching Survival analysis Panel data analysis Q2a. What effect does embedding a new CDSS in a community have on the attitudes and expectations of girls, boys, parents, and communities in the areas served by the new AMAA constructed new schools regarding: • Primary school retention and completion and transition to secondary schools; • Interest in continuing girls’ and boys’ education and/or vocational training; • Marriage and pregnancy; and • Future work/employment/career? Q2b. What are the attitudes and expectations of (1) girls and boys who have dropped out of school; and (2) girls and boys who are in the new CDSSs regarding: • Perceived value of attending school and education as experienced thus-far; • Future education; • Marriage and pregnancy; and • Future work/employment/ career Students Survey Parents expectations and attitudes survey Community leaders Students survey and dropouts’ interviews Repeated cross sections of students, and parents KII Longitudinal data students; tracking 80 students per school Gender & Education attitudes and expectations Gender & Education attitudes and expectations Balaka Treatment And Comparison Machinga and Balaka Treatment And Comparison May 2018 Mar 2021 May 2018 After drop out event Mar 2021 Difference in Difference Descriptive Analysis Descriptive analysis READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 53 Question Data Collection Activity Data Type/s Indicators Location and Group Date Analyses Q3. What are the reasons for girls and boys to: (1) drop out of school; (2) repeat; and (3) be held back? Dropout (repetition, etc.) interview Teacher interview Longitudinal data students; tracking 80 students per school Reasons for dropout, current situation Reasons for repetition, being held back Machinga and Balaka Treatment and Comparison Mar 2019 Mar 2020 Mar 2021 Regression analysis with controls and/or matching Panel Data Analysis Descriptive Analysis Q4. How are the CDSSs helping girls to overcome identified barriers, such as: education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school, and lack of space in secondary schools? Are there additional barriers to accessing schools faced by girls? Household interview for dropout girl students Parents and girls FGDs Longitudinal data students; tracking 80 students per school Qualitative Data Reasons for dropout. Current situation Challenges and advantages faced by parents and girls Machinga and Balaka Treatment Mar 2019 Mar 2020 Mar 2021 Mar 2020 Panel Data Descriptive Analysis Phenomenological analysis of Qualitative Data Q5a. What effect do the new CDSSs have on the perceptions of students regarding physical and socio-emotional aspects of safety? 5b. What are the perceptions of girls and boys dropping out of school regarding physical and socio-emotional aspects of safety? Interviews with students Dropout interview Students in the baseline sample still attending schools (Form 1 to 3) Students in the baseline sample that dropped out of school Student perception of safety around school/ going to school Perceptions of safety for those out of school Machinga and Balaka treatment and comparison Machinga and Balaka treatment and comparison Mar 2021 After dropout event and Mar 2021 Descriptive Analysis Descriptive Analysis Q6. What experience do new CDSSs have on attracting and retaining teachers? Teacher Roster Teacher Interviews Longitudinal Data Secondary School Teachers-ALL Turnover, job satisfaction, vacancies, etc. Machinga and Balaka Treatment Mar 2019 Mar 2020 Mar 2021 Descriptive analysis Q7. Are girls, their parents/guardians, and community members satisfied with the building facilities of the new schools? Students and parents’ surveys - School building satisfaction SUB-SAMPLE (30 students per secondary school, 10 in each Form) and their parents (25 approx.) Satisfaction with the school building Machinga and Balaka Treatment Mar 2021 Descriptive Analysis 54 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA METHODOLOGY OVERVIEW The evaluation and research design includes different methods depending on the question to be addressed. Some questions will be answered using quantitative data only, but other questions will require the use of qualitative information. In some cases, quantitative and qualitative data will complement each other to gain details and understanding. While for some questions it will be possible to identify causal effects using a quasi-experimental approach, other questions are more suitable for descriptive data analyses. The information to be used will be gathered from students, parents, teachers, and others in communities where the secondary schools were built (Treatment) and in communities that do not have secondary schools but are similar otherwise (Comparison). The sampling strategy can be found below. Below, we describe our approach to answering each of the research and evaluation questions. Question 1: What effect does embedding a new CDSS in a community have on (1) primary school completion and transition; and (2) secondary school enrollment, attendance, promotion and retention of girls and boys in the community? We will answer the question by comparing primary school completion rates, rates of transition to secondary school, attendance, promotion, and retention of students between treatment communities and comparison communities. We propose to do this in two ways: A) Using existing school administrative data (or creating the system to collect the necessary data) we will track the all the students (Census) attending standard 5, 6, 7, and 8 at the time of our baseline in May 2018. We will collect data yearly on these students, following those who transfer into the new secondary schools and create a panel to track them. These longitudinal data will allow us to calculate promotion and retention rates in secondary school, as well primary school completion rates. We could also learn which students take the primary school certification exam, and if they end up being admitted to secondary school. In addition, we will explore the feasibility of obtaining attendance data. This strategy allows us to assess the causal impact of school construction on the indicators of interest in question 1 and, given that administrative school data includes all students, it allows us to maximize the precision of our estimates. However, administrative data has shortcomings. Firstly, while it will be possible to use primary and secondary school administrative data in treatment communities, comparison communities will lack secondary school data. Secondly, administrative data does not include information about the students and their backgrounds other than considerably basic characteristics, such as sex. More detailed information will be needed to carry out our analysis. B) For a sample of students (Sample), in treatment and control communities, we will complement the tracking enrollment data with a baseline survey to collect information on socio-demographic characteristics of the student, disability status, socio-economic situation of his or her household, household contact information, and information on older siblings. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 55 The sample data will permit us to do several things: 1) use the household contact information to continue tracking students in comparison communities after they finish standard 8, allowing us to see their transition into secondary school and their promotion and retention in the years to follow; 2) use household contact information to reach students that drop out of schools (see Questions 3 and 4); and 3) understand how the educational path of boys and girls may differ depending on students’ individual circumstances, such as their age or household financial situation, something that cannot be done with just school administrative data. As Table 26 shows below, boys and girls that are in standard 8 at baseline could remain in the school system until the end of our study in March 2021, and could potentially reach form 3. Similarly, those students in standard 7 and 6, could potentially reach form 2 and 1, respectively, at endline. Students in standard 5 may be able to reach standard 8. Table 26. Potential Progress for Students May-18 Mar-19* Mar-20* Mar-21* Std 8 Form 1 Form 2 Form 3 Std 7 Std 8 Form 1 Form 2 Std 6 Std 7 Std 8 Form 1 Std 5 Std 6 Std 7 Std 8 * Tentative date To minimize attrition, the NORC team proposes, tracking the status of the students annually until the endline (planned for 2021). Analytically, to evaluate the impact of the program on, for example, enrollment for the average learner in a treatment community, we will compare enrollment rates between treatment and comparison communities at endline. The basic regression model we will use to estimate the causal impact of the program can be described by: 𝐸𝐸𝑐𝑐𝑐𝑐 = 𝛽𝛽0 + 𝛽𝛽1𝐷𝐷𝑐𝑐 + 𝛽𝛽2𝑆𝑆𝑐𝑐𝑐𝑐 + 𝑢𝑢𝑐𝑐𝑐𝑐 (1) where Eci is a binary variable that is 1 if learner i from community c is enrolled in school at endline and 0 if not; Dc is a binary variable indicating treatment status of community c; Sci is the standard that learner i is attending at baseline; uci is an error term; and 𝛽𝛽0, 𝛽𝛽1, and 𝛽𝛽2 are parameters to be estimated. The coefficient of interest is 𝛽𝛽1, which measures the effect of the school construction on enrollment at endline. The same model will be used to evaluate the impact of the program on other indicators. This model will, in principle, include all learners in the baseline roster in treatment and comparison groups, and will adapt to include the relevant subgroups depending on the indicator analyzed. For example, when estimating the impact of school construction on transition to secondary school, we will not include learners attending standard 5 at baseline because, in the best case, they would be attending standard 8 at endline. We expect to be able to identify each of the learners in the rosters as girls or boys, and therefore it will be possible to conduct the same analysis by sex. 56 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA For a sample of learners, we will also have additional information that includes their socio-demographic characteristics and their household information. For them, the equation (1) can include that information and therefore changes to: 𝐸𝐸𝑐𝑐𝑐𝑐 = 𝛽𝛽0 + 𝛽𝛽1𝐷𝐷𝑐𝑐 + 𝑋𝑋𝑐𝑐𝑐𝑐 ′ 𝛽𝛽2 + 𝑢𝑢𝑐𝑐𝑐𝑐 (2) where Xci is a vector of characteristics of learner i, such as age, sex, standard level at baseline, and household level characteristics such as assets. Again, the coefficient of interest is 𝛽𝛽1, which measures the effect of the school construction on enrollment at endline. This model therefore will include all learners in the sample, in treatment and comparison groups, and will control for their characteristics. Alternatively, a matching approach can be used. In addition to estimating the overall impact of the construction of secondary schools, we will also carry out a sub-group analysis to examine whether the impacts were different for sub-groups of interest, such as girls or poorer households. We can estimate similar models to measure the effect on other outcomes of interest, such as dropout rates, transition to secondary school rates, repetition rates, etc. It will be possible to estimate the effect on outcomes such as enrollment for each year, and to understand how the trends in treatment and comparison schools have evolved. Finally, it is also possible to use other analytical approaches such as survival analysis to estimate the effect of the school construction on the outcomes of interest. Survival analysis measures the time between the beginning of observation until the occurrence of an event. In the present case, the event could be dropping out of school, for example. NORC will evaluate the possibility of using this approach to analyze and communicate the findings. Question 2: (2a) What effect does embedding a new CDSS in a community have on the attitudes and expectations of girls, boys, parents, and communities in the areas served by the new AMAA constructed new schools regarding: • Primary school retention and completion and transition to secondary schools; • Interest in continuing girls’ and boys’ education and/or vocational training; • Marriage and pregnancy; and • Future work/employment/career? To answer this question, we will collect information from learners, parents, and community leaders about their attitudes towards gender norms and girls’ education, as well as their expectations regarding future education, work and employment, marriage, and family. Attitudes. To estimate the effects of secondary school construction on attitudes, the NORC team proposes to collect data on opinions at baseline from learners attending standard 5 to standard 8 and their parents, in treatment and comparison communities. The exact same data collection will take place READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 57 again at endline, returning to learners attending standard 5 to standard 8 and their parents. These cross￾section’ data will allow the use of a quasi-experimental designed to measure the impact of the secondary school construction activity on attitudes. We will estimate before-and-after changes in attitudes in communities that receive secondary schools and in communities that do not. The comparison of those changes is commonly known as the difference-in-difference (DiD) approach. The idea behind the DiD method is to eliminate the differences that the treatment and comparison groups may have (if any) and that are constant overtime. The methodology assumes that in the absence of the program, the two groups would display the same trends in attitudes.23 We will compare attitudes towards gender norms and girls’ education, among cross sections of learners (or parents) and estimate the causal impact of the school construction activity using regression models of the following form: 𝐴𝐴𝑐𝑐𝑐𝑐𝑐𝑐 = 𝛽𝛽0 + 𝛽𝛽1(𝐷𝐷𝑐𝑐 ∗ 𝑇𝑇𝑇𝑇𝑚𝑚𝑚𝑚) + 𝛽𝛽2𝐷𝐷𝑐𝑐 + 𝛽𝛽3𝑇𝑇𝑇𝑇 + 𝑋𝑋𝑐𝑐𝑐𝑐 ′ 𝛽𝛽4 + 𝑢𝑢𝑐𝑐𝑐𝑐𝑐𝑐 (3) where Acit is the attitude for learner i in community c, at time t, Dc is a binary variable indicating treatment status of community c; Time is a dummy variable equal to 1 at endline and 0 otherwise; Xci is a vector of characteristics of learner i, such as age, sex, standard level at baseline, and household level characteristics; ucit is an error term and 𝛽𝛽0, 𝛽𝛽1, 𝛽𝛽2 , 𝛽𝛽3, and the vector 𝛽𝛽4 are parameters to be estimated. The difference-in-difference estimator is 𝛽𝛽1, which measures the effect of the school construction on attitudes towards education, marriage and pregnancy, employment, etc. In addition, we will also conduct interviews with community leaders to collect information about their attitudes toward gender norms and girls’ education. The small number of interviews (12) will preclude us from an impact analysis; however, the information can be used to complete the understanding of the context. Expectations. Measuring the effect of the AMAA school construction on expectations only makes sense if we can measure expectations before the treatment group knows that a secondary school will be built in their community. Once it is known that a secondary school will be built, learners and parents would tend to adjust their expectations to the new reality that they will face, and this will tend to resemble expectations at endline in the treatment communities. The construction of schools in the district of Machinga is already taking place. Therefore, we will not try to measure the effect on expectations in that region. If the information about where school construction will take place in the Balaka district is still not public at the time of the baseline, we will measure changes in expectations there. We propose to collect data on expectations at baseline and endline from learners in standard 5 to standard 8 and their parents in the Balaka district and compute the DiD. However, if the information about school construction is already public in Balaka at the time of the baseline, we will use a qualitative approach to try to explore if and how expectations might have changed. In that case, we will collect information in Focus Groups Discussions (FGDs) with girl and boy learners, and parents. 23 This assumption can be checked using some of a learner’s background data collected to respond to evaluation question 1 (e.g., information about their older siblings). 58 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA If quantitative data is collected to measure expectations, we will run an equivalent regression model to the one used for attitudes. In this case, the dependent variable is an indicator related to expectations, rather than attitudes. 𝐸𝐸𝑐𝑐𝑐𝑐𝑐𝑐 = 𝛽𝛽0 + 𝛽𝛽1(𝐷𝐷𝑐𝑐 ∗ 𝑇𝑇𝑇𝑇 ) + 𝛽𝛽2𝐷𝐷𝑐𝑐 + 𝛽𝛽3𝑇𝑇𝑇𝑇 + 𝑋𝑋𝑐𝑐𝑐𝑐 ′ 𝛽𝛽4 + 𝑢𝑢𝑐𝑐𝑐𝑐𝑐𝑐 (4) where Ecit is the expectation for learner i in community c, at time t; Dc is a binary variable indicating treatment status of community c; Time is a dummy variable equal to 1 at endline and 0 otherwise; Xci is a vector of characteristics of learner i, such as age, sex, standard level at baseline, and household level characteristics; ucit is an error term and 𝛽𝛽0, 𝛽𝛽1, 𝛽𝛽2 , 𝛽𝛽3, and the vector 𝛽𝛽4 are parameters to be estimated. The difference-in-difference estimator is 𝛽𝛽1 which measures the effect of the school construction on attitudes towards education, marriage and pregnancy, employment, etc. It should be noted that the number of treatment and comparison communities in Balaka is small (13 in total) and, unless the changes in expectations are large, the estimates will tend to be imprecise. If this is the case, it will be noted and findings will not be taken as conclusive but rather suggestive of a particular trend. If instead we use qualitative data to learn about changes in expectations, we will use inductive thematic analysis to identify key subjects and recurring items or ideas emerging from FGDs with learners and parents. The emerged themes will be organized and preliminarily coded. Then, an iterative process will take place to identify the most common concepts and categories. (2b) What are the attitudes and expectations of (1) girls and boys who have dropped out of school; and (2) girls and boys who are in the new CDSSs regarding: • Perceived value of attending school and education as experienced thus-far; • Future education; • Marriage and pregnancy; and • Future work/employment/career? To answer part b of the question, we will again collect information from learners, parents and community leaders on attitudes and expectations from all the learners at baseline. Over time, we will update these opinions and aspirations of learners during new interviews. We will track all the learners included in the baseline sample over time. Each time a learner included in the baseline sample drops out of school, an interview will be triggered. Among other questions, we will include an update on their opinions and expectations. Dropouts will be again interviewed at endline. In the case of learners that stay in school, we will collect new information at endline. These data will allow us to see how learners’ attitudes and expectations change over time and to compare girls and boys that dropped out of school vs. those still attending secondary schools. The analysis will be descriptive and will not attempt to make any causal claim related to AMAA secondary schools. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 59 Questions 3 and 4: (3) What are the reasons for girls and boys to: (1) drop out of school; (2) repeat; and (3) be held back? (4) How are the CDSSs helping girls to overcome identified barriers, such as: education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school, and lack of space in secondary schools? Are there additional barriers to accessing schools faced by girls? Question 3 looks to identify the reasons girls and boys drop out of school, repeat levels, or are held back.24 We will focus also on risk factors and aspects that girls and parents describe as protecting girls’ capacities to attend school, persist from one grade to the next, and complete primary or secondary school. The goal of question 4 is to understand whether constructed secondary schools helps girls to overcome barriers like education costs, early pregnancy, early marriage, lack of WASH facilities, distance to secondary school, and lack of space in secondary schools. As in the case of question 2b, we will address this question by conducting interviews each time that one of the learners included in the sample created at baseline is found to have dropped out of school, repeats the standard, or is held back. In the cases of dropouts, a visit to the learner household would be necessary to find out the reasons behind the decision of leaving school and the current circumstances of the learner (e.g., marital status, activity, location, etc.). We will aim to document the main reason for dropping out and the surrounding circumstances, the current situation of the learner, and future plans. At endline, we will try to find these boys and girls, and ask again about their situation regarding education (e.g., if they went back to school), work and employment, marital status, children, and plans for the future. In cases of repetition or when the learner is held back, we will request information from the teacher involved in the decision. These learners will continue to be followed over time until endline. We will compile and analyze these data at endline and compare between the treatment and comparison communities to study if there are differences in the reasons learners drop out and to identify the main barriers girls face in attending secondary education. We will organize the data by reason for dropping out and we will also explore if there are differences in those reasons between boys and girls, and between the group of poorer learners and the rest. In addition, in the case of treatment communties, we will organize FGDs with girls and parents to discuss the advantages of having access to the new AMAA secondary school and also the oportunities for and challenges of staying (keeping learners) in schools. The data will be transcribed verbatim and an inductive thematic approach to analyize it will be followed to identify the main problems that can keep learners from attending school and the main advantages of having a secondary school close to their homes. The qualitative data will be coded and used to complement the quantitative data and gain depth in understanding. 24 By “held back,” we mean keeping the learners in standard 7, rather than promoting them to standard 8, in order to better prepare them for the final year of primary school and the certification test that takes place after completing the primary education level. 60 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Question 5: (5a) What effect do the new CDSSs have on the perceptions of learners regarding physical and socio￾emotional aspects of safety? To address the question, we will compare perceptions of safety between those attending secondary schools in their communities vs. those that attend a secondary school away from their community and need to travel or to relocate to other communities to access secondary education. We will compare secondary learners in treatment communities with secondary learners in comparison communities as we believe that the proximity to the school could reduce exposure to emotional, physical, and sexual violence. (5b) What are the perceptions of girls and boys dropping out of school regarding physical and socio￾emotional aspects of safety? For girls and boys who drop out of school, our approach to questions 3 and 4 partly answers this question since our interviews with this group will include questions of safety. We will also research how AMAA new secondary schools affect barriers such as distance to school. To learn more about the safety perceptions of dropouts, we will collect information during interviews triggered each time one of the learners in the baseline sample is found to have dropped out of school. In addition, at endline, we will revisit and re-interview dropouts and ask questions about safety from their counterparts who are still attending school. NORC will present a descriptive analysis of all these data about safety, making use of all the demographic and background data collected on these girls and boys at baseline. Question 6: What experience do new CDSSs have on attracting and retaining teachers? This question aims to document the experience that the new AMAA secondary schools have attracting and retaining teachers. NORC proposes to use administrative school data and teachers’ and head teachers’ interviews to construct longitudinal data on all new AMAA secondary school teachers. A descriptive analysis will be produced using that data. We will capture teachers’ basic demographic and education characteristics and document school vacancies. Over time, we will complete the information by adding data on teacher turnover and replacements, seniority, working conditions, and job satisfaction, etc. Teacher interviews will also include questions to document the main positive and negative aspects of teachers’ experiences in the new schools. Findings from research question 7, which relate to the satisfaction with the school building, will also be included. We will create the baseline for the teacher panel in the second visit to treatment communities (currently expected to take place in March 2019) when all secondary schools are supposed to be finished and running. The information will be updated annually in 2020 and 2021 during the same visits when we will update the panel of learners. Obviously, in this case, the data collection will only take place in treatment communities. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 61 It should be noted that the proposed approach consists of documenting and trying to understand the experiences of teachers in the rural locations where they will be deployed. We will not try to compare their retention or turnover rates with those of teachers in other schools. Question 7: Are girls, their parents/guardians, and community members satisfied with the construction design and facilities of the new schools? The construction design attributes should be further specified in collaboration with USAID. This question seeks to understand learners,’ parents,’ and community leaders’ satisfaction with the new AMAA secondary school buildings. The AMAA buildings follow the Malawi MoEST standards, therefore the aim is not to inquire about AMAA schools as a distinctive group but generally about the regular schools offered by the public sector. We will interview a random sample of AMAA secondary school learners, parents, and teachers at endline in 2021. During these interviews, we will inquire about their satisfaction with the building structure, including classrooms, assembly hall, laboratory, library, fences and gates, latrines, outdoor space, and water sources, etc. The analysis of the information will be descriptive, but we will use the findings from other questions, such as reasons for dropping out of school or discussions related to question 4, to complement it. SAMPLING STRATEGY AND SELECTION OF RESPONDENTS As with any quantitative analysis based on survey data, the required sample size is determined by a mathematical calculation that depends on a number of factors. These include features of the study design, properties of the data and outcome variables, etc. In our case, the main parameter was the number of treatment communities that is given by the number of schools that are being built. We calculated the minimum effect that will be possible to detect with 11 treatment communities, making the following assumptions: At baseline, we created a sample of learners attending standard 5, 6, 7, and 8 in each of the treatment and comparison communities. We randomly selected 20 of the enrolled learners from each standard. The sample included 10 girls and 10 boys whenever possible. ■ α is the significance level of the test, or probability of Type I error. We use the standard value of 0.05. ■ β is the power of test, where (1-β) is the probability of Type II error. We use the standard value of 0.8. ■ ρ is the intracluster correlation coefficient, or ICC. The ICC in the present case is a measure of how much variability lies between schools and how much lies within schools. We assume an ICC of 0.1. ■ r2 is the proportion of the variation in the outcome due to the covariates anticipated in the regression analysis. In our case, these covariates will include a range of household and individual characteristics. For this parameter, we assume an approximate value of 0.3 62 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA ■ Attrition is assumed to be 20% at endline. ■ The number of treatment and control communities is identical: eleven in each group. ■ A random sample of eighty learners per community (20 per standard). Under those assumptions, we calculated the minimum detectable effect size (MDES). The MDES is the smallest impact of the activity on the outcome variable that the evaluation will be able to detect. We estimated that we will be able to detect a change in transition to secondary school from 30% to 50% when we do the analysis for each standard separately, and from 30% to 47% when we analyze standard 6 to 8 altogether. In addition, we will be able to detect changes from 50% to 70% in completion of primary school when analyzing standards separately, and from 50% to 65% when analyzing all standards together. Summarizing, the sample includes twenty-two schools (includes treatment and comparison) with eighty learners in each school, for a total sample of 1,760 learners. In addition, we created a sample of the learners’ parents/caregivers to interview. We estimated that around 25-35 parents will respond our interview. We also interviewed the community leaders to learn about their attitudes and expectations.25 The teacher and head teacher information will be censual and therefore will include all AMAA new secondary school teachers and head teachers. We will also collect qualitative data during focus group discussions. In March 2020, we will randomly select four treatment communities and conduct 2 FGDs with girl learners and 2 FGDs with parents in each of them. Girl learners and parents will be selected randomly from all those eligible.26 Finally, in March 2021 we will interview a random sample of learners in form one to form three attending the new AMAA secondary schools and their parents. The sample will include thirty learners per school, ten in each form (half of them girls whenever possible), for a total of three hundred learners. We will also invite thirty parents to respond to the school building satisfaction survey. These samples assume a confidence level of 95% and a margin of error of 5%. LIMITATIONS The evaluation team has identified some limitations inherent to the design of this evaluation. We list some of the more relevant limitations below: Representativeness of the Sample. The sample is representative of the areas where AMAA school construction will take place, in the areas of Machinga and Balaka districts. Therefore, results are not directly generalizable at the national level, other geographical areas, or to other school levels. Despite this, we hope the study will be informative and contribute to the evidence regarding secondary school construction, secondary school attendance, and dropouts in developing countries. Longitudinal Attrition and Absenteeism. The evaluation employs a longitudinal design where we will interview the same learners and teachers at baseline, follow-ups, and endline. Some learners will be still 25 According to a review conducted by Guest et al (2017) 12 interviews are enough to discover between 80 and 90 percent of the themes. 26 Previous research shows that conducting six FGD (by subpopulation of interest) allows discovering between 80 and 90 percent of the themes (Guest et al, 2017). READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 63 attending school, while some other will drop out. Absenteeism rates are, however, high and could present a problem. For dropouts, we plan to visit them at home. All measures available to keep track of learners are in place to reduce attrition. School Sample Size. It should be noted that the nature of the activity necessarily limits our sample size. The number of treatment and comparison communities in Balaka is small (13 in total) and unless the changes are large, estimates will tend to be imprecise. If this is the case, it will be noted and findings will not be taken as conclusive but rather suggestive of a particular trend. In general, however, we expect the sample to be large enough to estimate the expected effects. 64 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA ANNEX III: VIOLENCE INDICATORS Violence Indicator Questions Emotional Violence by Adult Has a parent, adult caregiver or other adult relative ever: a. told you that you were not loved, or did not deserve to be loved? b. said they wished you had never been born or were dead? c. ridiculed you or put you down, for example say that you were stupid or useless? Physical Violence by Peer or Adult Has a person your own age ever: a. punched, kicked, whipped, or beat you with an object? b. choked, smothered, tried to drown you, or burned you intentionally? c. used or threatened you with a knife, gun, or other weapon? Has a parent, adult caregiver, or other adult relative ever: a. punched, kicked, whipped, or beat you with an object? b. choked, smothered, tried to drown you, or burned you intentionally? c. used or threatened you with a knife, gun, or other weapon? Has an adult in your community (not the people we discussed before) ever? a. punched, kicked, whipped, or beat you with an object? b. choked, smothered, tried to drown you, or burned you intentionally? c. used or threatened you with a knife, gun, or other weapon? Transactional Sex Have you ever received money, food, favors or any gifts in exchange for sex? Sexual Violence Has anyone ever touched you in a sexual way without your permission, but did not try and force you to have sex? Has anyone ever tried to make you have sex against your will but did not succeed? Has anyone ever physically forced you to have sex and did succeed? Witness Community Violence Outside of your home and family environment, how many times did you see anyone get attacked? Witness Family Violence How many times did you see or hear your parent punched, kicked, or beaten up by your other parent, or their boyfriend or girlfriend? How many times did you see or hear a parent punch, kick, or beat your brothers or sisters? READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 65 ANNEX IV: ADDITIONAL ANALYSIS Table 27. Main Outcome Indicators, by Student Gender Outcome Overall Pct. Girls Pct. Pct. Boys Diff. Student is a dropout at endline 38.7 42.3 34.9 7.5 *** Standard 5 39.6 44.5 34.5 10.0 ** Standard 6 41.4 45.5 37.1 8.5 * Standard 7 42.0 44.1 40.0 4.1 Standard 8 31.4 35.0 27.7 7.3 Student was ever a dropout 39.7 43.3 36.0 7.3 *** Standard 5 40.9 45.8 35.8 9.9 ** Standard 6 42.3 45.5 38.8 6.7 Standard 7 42.7 45.4 40.0 5.4 Standard 8 32.7 36.3 29.1 7.2 Student is still enrolled in school at endline 61.3 57.7 65.1 -7.5 *** Standard 5 60.4 55.5 65.5 -10.0 ** Standard 6 58.6 54.5 62.9 -8.5 * Standard 7 58.0 55.9 60.0 -4.1 Standard 8 68.6 65.0 72.3 -7.3 Student is still in primary school at endline 25.6 24.1 27.0 -2.9 Standard 5 60.4 55.5 65.5 -10.0 ** Standard 6 33.6 32.3 34.8 -2.5 Standard 7 6.6 6.6 6.7 -0.1 Standard 8 0.2 0.0 0.5 -0.5 Student is in secondary school at endline 48.0 45.1 51.0 -5.9 ** Standard 5 n/a n/a n/a n/a Standard 6 25.1 22.1 28.1 -6.0 Standard 7 51.3 49.3 53.3 -4.0 Standard 8 68.4 65.0 71.8 -6.8 Ever made it to secondary school 52.4 49.5 55.3 -5.8 ** Standard 5 n/a n/a n/a n/a Standard 6 25.1 22.1 28.1 -6.0 Standard 7 55.1 52.0 58.2 -6.2 Standard 8 77.9 75.8 80.0 -4.2 Moved to Attend Secondary School 37.6 38.2 37.1 1.1 Standard 5 n/a n/a n/a n/a Standard 6 16.7 20.0 14.3 5.7 Standard 7 32.9 33.3 32.4 0.9 Standard 8 42.0 42.9 41.4 1.5 Student ever repeated a standard 37.3 35.2 39.4 -4.3 * Standard 5 46.3 43.6 49.1 -5.5 66 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Overall Girls Boys Outcome Pct. Pct. Pct. Diff. Standard 6 45.5 42.6 48.7 -6.1 Standard 7 35.0 33.5 36.4 -3.0 Standard 8 21.7 20.2 23.2 -3.0 Student ever took PSLCE Exam 61.4 60.0 62.8 -2.7 Standard 5 23.4 19.5 27.4 -7.9 ** Standard 6 52.7 54.0 51.3 2.7 Standard 7 73.5 70.0 76.9 -6.8 * Standard 8 97.7 99.1 96.4 2.7 * Student ever took PSLCE and obtained passing score 53.5 50.5 56.6 -6.2 *** Standard 5 19.5 16.1 23.0 -6.9 * Standard 6 44.4 45.1 43.8 1.4 Standard 7 63.9 59.0 68.9 -9.9 ** Standard 8 87.8 83.9 91.8 -8.0 ** *** p < 0.01, ** p < 0.05, * p < 0.1. All statistics calculated using entropy weights for statistical balancing between treatment and comparison groups based on baseline characteristics. Table includes all tracked students for whom enrollment status at endline could be determined. The tracked sample included 1823 students, and endline enrollment status could be determined for 1816. The 1816 students include 462 in Standard 5 at baseline, 459 in Standard 6, 452 in Standard 7, and 443 in Standard 8. 1354 students had the potential to make it to secondary school by endline (i.e., were in Standard 6 or higher at baseline). Table 28. Difference in Difference Estimation of Aspirations and Expectations – Machinga and Balaka Districts (1) (2) (3) (4) VARIABLES Highest desired level of education is college/universit y Expects to reach desired level of education Where on the ladder do you feel you personally stand at the present time? Where on the ladder will you be in the future? Year = 2021 5.411* -7.618** 0.414** -0.707*** (2.627) (3.128) (0.157) (0.152) Treatment 2.039 -2.848 -0.121 0.0378 (2.071) (3.266) (0.183) (0.133) Year = 2021 * Treatment -2.292 -5.417 0.218 0.0356 (3.335) (6.032) (0.187) (0.169) Constant 84.84*** 69.94*** 3.435*** 7.043*** (1.481) (2.835) (0.148) (0.0964) Observations 2,466 2,463 3,037 3,027 R-squared 0.004 0.013 0.017 0.032 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 67 Table 29. Difference in Difference Estimation of Aspirations and Expectations – Balaka District (1) (2) (3) (4) VARIABLES Highest desired level of education is college/universi ty Expects to reach desired level of education Where on the ladder do you feel you personally stand at the present time? Where on the ladder will you be in the future? Year = 2021 2.550 -11.10*** 0.128 -0.729*** (3.008) (2.653) (0.177) (0.230) Treatment 1.532 -2.010 -0.217 0.00506 (2.758) (4.892) (0.253) (0.160) Year = 2021 * Treatment -1.304 -3.688 0.371 0.0784 (4.577) (8.693) (0.216) (0.251) Constant 84.34*** 69.89*** 3.654*** 7.101*** (1.276) (4.043) (0.200) (0.126) Observations 1,393 1,391 1,713 1,707 R-squared 0.001 0.016 0.008 0.033 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 68 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA ANNEX V: BALANCE BETWEEN TREATMENT AND CONTROL GROUPS Table 30. Balance between Treatment and Control (1) (2) (1)-(2) Treatment Comparison Difference Variable Mean/SE Mean/SE Student Grade = Standard 5 0.254 0.255 -0.001 [0.004] [0.003] Student Grade = Standard 6 0.255 0.250 0.005 [0.004] [0.002] Student Grade = Standard 7 0.253 0.247 0.006 [0.004] [0.003] Student Grade = Standard 8 0.238 0.249 -0.011 [0.010] [0.002] Student is repeating the grade level 0.213 0.192 0.021 [0.027] [0.019] How old are you? 14.468 14.250 0.218 [0.097] [0.118] Student Sex = Female 0.500 0.514 -0.014 [0.011] [0.006] Mother's Education = None 0.150 0.152 -0.001 [0.030] [0.024] Mother's Education = Some primary 0.610 0.551 0.059 [0.020] [0.029] Mother's Education = Complete primary 0.126 0.146 -0.020 [0.014] [0.015] Mother's Education = Some secondary 0.093 0.105 -0.012 [0.017] [0.022] Mother's Education = Complete secondary 0.014 0.032 -0.018* [0.006] [0.008] Mother's Education = Higher than secondary 0.006 0.011 -0.005 [0.003] [0.004] Mother's Education = Other 0.002 0.004 -0.003 [0.001] [0.002] On-Track Education (according to age-for￾grade schedule) 0.187 0.231 -0.044* [0.014] [0.021] Student missed at least 1 day of school last week 0.280 0.279 0.001 [0.030] [0.023] How many week? days did you miss school last 0.521 0.555 -0.034 [0.065] [0.049] How long does it take you to travel from home to school, in minutes? 45.944 44.628 1.316 [2.877] [1.699] Did you eat anything today? 0.364 0.397 -0.032 [0.041] [0.039] Asset Index 1.799 1.973 -0.174 [0.118] [0.070] 0.869 0.848 0.020 READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 69 (1) (2) (1)-(2) Treatment Comparison Difference Variable Mean/SE Mean/SE Student would like to continue to university after secondary school [0.015] [0.015] Think about yourself at age 25, would you like to have job or be working at that 0.933 0.932 0.001 [0.012] [0.011] Would you like to get married at some point in your life? 0.889 0.883 0.006 [0.030] [0.016] At what approximate age would you like to get married? 28.372 28.531 -0.159 [0.191] [0.170] Would you like to have children at some point in your life? 0.900 0.897 0.003 [0.026] [0.014] At what approximate age would you like to have your first child? 30.420 30.737 -0.317 [0.254] [0.219] Where on the ladder do you feel you personally stand at the present time? 3.314 3.435 -0.121 [0.110] [0.151] Where on the ladder do you think you will be at age 25? 7.081 7.043 0.038 [0.094] [0.098] Student Gender attitudes Index (0=Biased; 1=Equal) 0.532 0.542 -0.011 [0.006] [0.007] Level 1 Disability 0.437 0.467 -0.030 [0.027] [0.019] Level 2 Disability 0.149 0.172 -0.023 [0.021] [0.018] Level 3 Disability 0.023 0.016 0.007 [0.006] [0.004] Caregiver's Sex = Female 0.728 0.756 -0.028 [0.030] [0.022] What is your age? 40.886 41.373 -0.487 [0.665] [0.570] Student Sex = Female 0.495 0.503 -0.008 [0.010] [0.008] Student Grade = Standard 5 0.248 0.255 -0.007 [0.005] [0.003] Student Grade = Standard 6 0.250 0.246 0.004 [0.005] [0.005] Student Grade = Standard 7 0.255 0.251 0.004 [0.004] [0.005] Student Grade = Standard 8 0.248 0.248 -0.001 [0.005] [0.004] Caregiver's Education = None 0.202 0.188 0.015 [0.029] [0.028] Caregiver's Education = Some primary 0.537 0.533 0.005 [0.016] [0.027] Caregiver's Education = Complete primary 0.104 0.131 -0.027 [0.016] [0.015] Caregiver's Education = Some secondary 0.096 0.092 0.005 70 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA (1) (2) (1)-(2) Treatment Comparison Difference Variable Mean/SE Mean/SE [0.015] [0.021] Caregiver's Education = Complete secondary 0.041 0.039 0.002 [0.010] [0.008] Caregiver's Education = Higher than secondary 0.017 0.013 0.004 [0.008] [0.005] Caregiver's Education = Other 0.002 0.004 -0.002 [0.002] [0.004] Caregiver = Parent 0.802 0.769 0.034 [0.025] [0.020] Caregiver = Grandparent 0.087 0.098 -0.012 [0.018] [0.013] Caregiver = Aunt/Uncle 0.055 0.046 0.010 [0.011] [0.007] Caregiver = Sibling 0.055 0.074 -0.019 [0.011] [0.011] Do you know how to write and read? 0.542 0.575 -0.033 [0.037] [0.039] Caregiver Gender attitudes Index (0=Biased; 1=Equal) 0.505 0.502 0.003 [0.013] [0.008] Caregiver would like student to continue to university after secondary school 0.826 0.800 0.026 [0.022] [0.026] Think about [ChildName] at 25, would you like her/him to hold a job or work? 0.949 0.950 -0.000 [0.015] [0.014] Where on the ladder do you feel you personally stand at the present time? 2.687 2.709 -0.022 [0.103] [0.118] Where do you think you will be on the ladder in four years from now? 4.337 4.397 -0.061 [0.117] [0.128] Where do you think that [ChildName] will be when he/she is your age? 6.419 6.529 -0.111 [0.092] [0.116] Age you expect [ChildName] to start earning money to support your household? 25.965 25.917 0.047 [0.200] [0.380] Age you expect [ChildName] to leave full￾time education? 23.410 22.661 0.749 [0.377] [0.735] Age you expect [ChildName] to be financially independent of HIS/HER parents? 27.806 28.222 -0.416 [0.263] [0.319] Age you expect [ChildName] to leave this household? 28.796 28.774 0.021 [0.297] [0.365] Age you expect [ChildName] to get married and start living together with a husband/wife? 29.799 30.103 -0.304 [0.271] [0.270] Age you expect [ChildName] to have a child? 30.982 31.142 -0.161 [0.301] [0.298] Notes: The value displayed for t-tests are the differences in the means across the groups. Standard errors are clustered at community level ***, **, and * indicate significance at the 1, 5, and 10 percent critical level. READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA | 71 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Alicia Menendez Title Principal Investigator Organization NORC at the University of Chicago Evaluation Position? ☒Team Leader ☐Team member Evaluation Award Number Cooperative Agreement NoAID-OAA-A-17-00003 USAID Project(s) Evaluated The Apatseni Mwayi Atsikana Aphunzire (AMAA) Evaluation, Malawi I have real or potential conflicts of interest to disclose. ☐Yes ☒No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 19, 2018 72 | READING AND ACCESS EVALUATION AND RESEARCH FINAL REPORT USAID.GOV APATSENI MWAYI ATSIKANA APHUNZIRE AMAA Name Gregory Lee Haugan Title Principal Research Analyst Organization NORC at the University of Chicago Evaluation Position? ☐Team Leader ☒ Team member Evaluation Award Number (contract or other instrument) Cooperative Agreement NoAID-OAA-A-17-00003 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) The Apatseni Mwayi Atsikana Aphunzire (AMAA) Evaluation, Malawi I have real or potential conflicts of interest to disclose. ☐Yes ☒ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 03/19/2018 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523