EVALUATION THE MALAWI GIRLS’ EMPOWERMENT THROUGH EDUCATION AND HEALTH ACTIVITY (ASPIRE): 2017 PERFORMANCE EVALUATION REPORT February 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by EnCompass LLC under the Evaluation Services IDIQ, Task Order AID-612-TO-17- 00002. Prepared for the United States Agency for International Development USAID Contract Number AID-OAA-I-15-00021 Task Order AID-612-TO-17-00002 February 15, 2018 Implemented by: EnCompass LLC 1451 Rockville Pike, Suite 600 Rockville, MD 20852 Phone: +1 301-287-8700 Fax: +1 301-685-3720 www.encompassworld.com THE MALAWI GIRLS’ EMPOWERMENT THROUGH EDUCATION AND HEALTH ACTIVITY (ASPIRE) 2017 PERFORMANCE EVALUATION REPORT DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. Policy, Planning, and Learning–Learning, Evaluation, and Research This task order is implemented through the Policy, Planning, and Learning–Learning, Evaluation, and Research (PPL-LER) indefinite delivery, indefinite quantity contract, funded by the United States Agency for International Development (USAID). USAID’s Bureau for Policy, Planning, and Learning awarded EnCompass LLC the 5-year PPL-LER contract to provide technical and advisory services for performance and impact evaluations, evaluation and performance monitoring capacity building, and performance monitoring activities at the Mission, Bureau, and Agency-wide levels. PPL-LER task orders design and implement evaluation studies and assessments based on rigorous evidence sources (quantitative and qualitative), develop and deliver evaluation and performance monitoring training, and provide technical assistance in performance monitoring for USAID development programs worldwide. Recommended Citation Falconer-Stout, Z., K. Simmons, P. Godbole, S. Topolansky, R. Frischkorn, and L. Franco. 2018. The Malawi Girls’ Empowerment through Education and Health Activity (ASPIRE): 2017 Performance Evaluation Report. Prepared for the United States Agency for International Development. Rockville, MD: EnCompass LLC. Acknowledgments This evaluation was led by Nick Shawa and managed by Zachariah Falconer-Stout, with team members Maureen Chirwa, Lynne Franco, Rebecca Frischkorn, Pragati Godbole, Abigail Kazembe, Kelsey Simmons, and Sabine Topolansky. The University of Malawi Centre for Social Research, subcontractor to EnCompass, was responsible for quantitative data collection, under the management of Nick Shawa and Maxton Tsoka, and for supporting qualitative data collection, under the management of Maureen Chirwa and Abigail Kazembe. Armando Levinson served as sampling statistician. Crystal Cason created the data visualizations. Jaime L. Jarvis and Jelena Burgić Simmons edited the report. The evaluation team gratefully acknowledges the input provided by the ASPIRE activity team at Save the Children and its subcontractors, and the financial support of USAID. The team is also grateful to the respondents and beneficiaries whose input made this evaluation possible, and to the Government of Malawi. February 2018 | ASPIRE Performance Evaluation Report i CONTENTS LIST OF EXHIBITS ............................................................................................................II ACRONYMS ...................................................................................................................... IV EXECUTIVE SUMMARY .................................................................................................. V 1. EVALUATION PURPOSE AND QUESTIONS ....................................................... 1 2. BACKGROUND ........................................................................................................... 3 2.1 Context .............................................................................................................................................................. 3 2.2 ASPIRE Design and Theory of Change ....................................................................................................... 4 2.3 ASPIRE Interventions ...................................................................................................................................... 6 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS...............................10 3.1 Evaluation Design ..........................................................................................................................................10 3.2 Data Collection Tools and Methods.........................................................................................................11 3.3 Sampling Design .............................................................................................................................................12 3.4 Analysis ............................................................................................................................................................13 3.5 Limitations .......................................................................................................................................................14 4. DESCRIPTION OF SAMPLE ....................................................................................15 4.1 Characteristics of Head Teachers in the Sample ...................................................................................16 4.2 Characteristics of Primary School Student Respondents ....................................................................17 4.3 Characteristics of Secondary School Student Respondents ...............................................................18 5. FINDINGS ...................................................................................................................20 5.1 Evaluation Question 1: ASPIRE’s Progress..............................................................................................20 5.2 Evaluation Question 2: Integration with other USAID-Funded Activities .......................................52 5.3 Evaluation Question 3: Coordination with District Government .....................................................58 5.4 Evaluation Question 4: Engagement with Private Sector .....................................................................60 5.5 Evaluation Question 5: Accomplishments, Best Practices, and Lessons Learned ..........................63 5.6 Evaluation Question 6: Beneficiaries’ Recommendations on Adaptations ......................................69 6. CONCLUSIONS AND RECOMMENDATIONS ...................................................79 6.1 Conclusions ....................................................................................................................................................79 6.2 Recommendations .........................................................................................................................................84 REFERENCES ....................................................................................................................89 ANNEX 1: EVALUATION SCOPE OF WORK ............................................................92 ANNEX 2: DETAILED DESIGN, METHODS, AND LIMITATIONS ........................99 ANNEX 3: DATA TABLES ...........................................................................................110 ANNEX 4: DOCUMENTS REVIEWED .......................................................................143 ANNEX 5: STATEMENTS OF DIFFERENCE ............................................................145 ANNEX 6: CONFLICT OF INTEREST DISCLOSURES ...........................................146 ANNEX 7: BACKGROUND ON EVALUATION TEAM MEMBERS ......................157 ANNEX 8: DATA COLLECTION TOOLS ................................................................. 163 February 2018 | ASPIRE Performance Evaluation Report ii LIST OF EXHIBITS EXHIBIT 1: EVALUATION QUESTIONS .............................................................................................................................. 1 EXHIBIT 2: ASPIRE FUNDING STREAMS ........................................................................................................................... 4 EXHIBIT 3: ASPIRE INTERVENTIONS BY DISTRICT.......................................................................................................... 5 EXHIBIT 4: ASPIRE RESULTS FRAMEWORK...................................................................................................................... 6 EXHIBIT 5: QUANTITATIVE SAMPLE OVERVIEW ............................................................................................................15 EXHIBIT 6: QUALITATIVE SAMPLE BY RESPONDENT TYPE, SEX, AND METHOD.........................................................15 EXHIBIT 7: QUALITATIVE SAMPLE BY DISTRICT, SCHOOL TYPE, AND SETTING ........................................................16 EXHIBIT 8: TOTAL NUMBER OF GROUP INTERVIEW AND GROUP DISCUSSION RESPONDENTS .............................16 EXHIBIT 9: NUMBER OF YEARS SPENT AS HEAD TEACHER OR DEPUTY HEAD TEACHER .........................................16 EXHIBIT 10: HIGHEST ACADEMIC QUALIFICATION FOR TEACHERS...........................................................................16 EXHIBIT 11: AGE DISTRIBUTION OF STUDENTS IN PRIMARY GRADES........................................................................17 EXHIBIT 12: MAIN LANGUAGES SPOKEN AT HOME AND WITH FRIENDS (PRIMARY STUDENTS) ............................18 EXHIBIT 13: AGE DISTRIBUTION OF STUDENTS IN SECONDARY GRADES................................................................. 18 EXHIBIT 14: MAIN LANGUAGES SPOKEN AT HOME AND WITH FRIENDS (SECONDARY STUDENTS) .....................19 EXHIBIT 15: GIRLS’ PASS RATES IN NATIONAL EXAMINATIONS (INDICATOR 4)......................................................21 EXHIBIT 16: DROPOUT AND REPETITION RATES, STANDARDS 4–8 ..........................................................................22 EXHIBIT 17: ASPIRE SUPPORT FOR LITERACY INSTRUCTION (OUTPUT 1)..............................................................25 EXHIBIT 18: ASPIRE SUPPORT FOR SCHOOL-BASED HEALTH ACTIVITIES (OUTPUT 2)..........................................25 EXHIBIT 19: ASPIRE SUPPORT FOR SCHOOL WASH (OUTPUT 3)...........................................................................25 EXHIBIT 20: ASPIRE SUPPORT TO REDUCE STRUCTURAL AND CULTURAL BARRIERS (OUTPUT 3)......................26 EXHIBIT 21: ENGLISH ORAL READING DISTRIBUTION, OVERALL AND DISAGGREGATED BY GRADE.....................28 EXHIBIT 22: CHICHEWA ORAL READING DISTRIBUTION, OVERALL AND DISAGGREGATED BY GRADE................29 EXHIBIT 23: PERCENTAGE OF STUDENTS WHO ACCURATELY COMPLETED ENGLISH READING COMPREHENSION TASKS........................................................................................................................................................................30 EXHIBIT 24: MEAN ENGLISH READING COMPREHENSION (DISAGGREGATED BY SEX) ............................................30 EXHIBIT 25: PERCENTAGE OF STUDENTS WHO ACCURATELY COMPLETED CHICHEWA READING COMPREHENSION TASKS........................................................................................................................................31 EXHIBIT 26: MEAN CHICHEWA READING COMPREHENSION (DISAGGREGATED BY SEX) ......................................31 EXHIBIT 27: PERCENTAGE OF STUDENTS WHO ACCURATELY COMPLETED ENGLISH LISTENING COMPREHENSION TASKS........................................................................................................................................31 EXHIBIT 28: MEAN ENGLISH LISTENING COMPREHENSION (DISAGGREGATED BY SEX) ..........................................31 February 2018 | ASPIRE Performance Evaluation Report iii EXHIBIT 29: ASPIRE LITERACY DATA............................................................................................................................32 EXHIBIT 30: PERCENT OF STUDENTS WHO HAVE HEARD OF HIV AND AIDS .........................................................34 EXHIBIT 31: PERCENT OF STUDENTS WHO INDICATED WHETHER HIV/AIDS IS TRANSMITTED THROUGH: ......34 EXHIBIT 32: PERCENT OF STUDENTS WHO STATED IT IS POSSIBLE TO REDUCE RISK OF HIV TRANSMISSION BY: ..................................................................................................................................................................................35 EXHIBIT 33: PERCENT OF STUDENTS WHO CORRECTLY REJECTED COMMON MISCONCEPTIONS THAT IT IS POSSIBLE TO TRANSMIT HIV/AIDS BY: ................................................................................................................37 EXHIBIT 34: PERCENT OF STUDENTS WHO HAVE ACCESSED SEXUAL AND REPRODUCTIVE HEALTH (SRH) SERVICES VIA THE FOLLOWING: ............................................................................................................................39 EXHIBIT 35: SATISFACTION WITH YOUTH-FRIENDLY HEALTH SERVICES REFERRAL...............................................41 EXHIBIT 36: OF THOSE WHO REPORTED KNOWING OF CONTRACEPTIVE METHODS, WHAT TYPES DID THEY KNOW ABOUT? (FIVE MOST COMMON) ..............................................................................................................42 EXHIBIT 37: CONDOM USE AMONG SEXUALLY ACTIVE STUDENTS...........................................................................44 EXHIBIT 38: ASPIRE’S SUPPORT FOR HYGIENE AND SANITATION ............................................................................45 EXHIBIT 39: PERCENTAGE OF GIRLS WHO COME TO SCHOOL WHEN MENSTRUATING .......................................45 EXHIBIT 40: PERCENT OF STUDENTS WHO FEEL COMFORTABLE USING SCHOOL TOILETS....................................45 EXHIBIT 41: SCHOOLS OPERATIONALIZING STRATEGIES TO SAFEGUARD GIRLS .....................................................48 EXHIBIT 42: ASPIRE’S INTEGRATION WITH USAID PARTNERS.................................................................................52 EXHIBIT 43: RESOURCES ASPIRE HAS GENERATED FROM PRIVATE SECTOR ENGAGEMENT AS OF FY 2017 QUARTER 2 .............................................................................................................................................................61 EXHIBIT 44: PRIVATE SECTOR ENGAGEMENT AS OF FY 2017, QUARTER 2 .............................................................61 EXHIBIT 45: WHERE STUDENTS INDICATED HAVING HEARD ABOUT HIV/AIDS ....................................................68 EXHIBIT 46: COMPONENT SKILLS IN READING COMPREHENSION AND THEIR STRUCTURAL RELATIONSHIPS.....82 EXHIBIT 47: STUDENT SAMPLE BY GRADE ................................................................................................................. 105 February 2018 | ASPIRE Performance Evaluation Report iv ACRONYMS 3-C Co-locating, coordinating, and collaborating MSCE Malawi School Certificate of Education ASPIRE The Malawi Girls’ Empowerment through Education and Health Activity MIE Malawi Institute of Education BLM Banja la Mtsogolo MoEST Ministry of Education, Science, and Technology CDCS Country Development Cooperation Strategy NGO Nongovernmental organization CRECCOM Creative Centre for Community Mobilisation PEPFAR U.S. President’s Emergency Plan for AIDS Relief DEM District Education Management PSI Population Services International DEMIS District Education Management Information System PSLCE Primary School Leaving Certificate Examination DHS Demographic Health Survey DREAMS Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe PTA Parent-teacher association EGRA Early Grade Reading Activity R Respondent (followed by a respondent number for direct quotes; e.g., R1) EMIS Education Management Information System SEGREM Strengthening Early Grade Reading in Malawi FAWEMA Forum for African Women Educationalists SIFPO2 Support for International Family Planning Organizations 2 FY Fiscal year VCT Voluntary counseling and testing IUD Intrauterine device UNICEF United Nations Children’s Fund KAP Knowledge, attitudes, and practices (survey) USAID United States Agency for International Development MERIT Malawi Early Grade Reading Improvement Activity WASH Water, sanitation, and hygiene WHO World Health Organization February 2018 | ASPIRE Performance Evaluation Report v EXECUTIVE SUMMARY PURPOSE AND EVALUATION QUESTIONS This external performance evaluation of the Malawi Girls’ Empowerment through Education and Health Activity (ASPIRE), conducted 2.5 years after ASPIRE began, establishes the activity’s progress against its objectives, proposes adaptations for the final year, and captures lessons for application in future girls’ empowerment, health, and education programming in Malawi. The primary audience for this evaluation is the United States Agency for International Development (USAID) Malawi’s Education and Health, Population, and Nutrition offices, including the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) team. ASPIRE implementing partners and implementing offices of the Government of Malawi are also expected to benefit from evaluation results. PROJECT BACKGROUND ASPIRE is a 4-year, $16.2 million, USAID activity implemented by Save the Children and three partners, designed to support the Government of Malawi to improve girls’ achievement in upper primary and secondary school, and ultimately, girls’ empowerment. ASPIRE has three outputs:  Output 1: Reading skills for girls in upper primary school improved  Output 2: Adoption of positive sexual and healthcare-seeking behaviors among youth ages 10–19  Output 3: Key structural and cultural barriers for girls ages 10–19 decreased. In its first 2.5 years, ASPIRE conducted a range of activities, engaging a variety of actors to confront barriers that impede girls’ achievement. ASPIRE also employs strategies aligned with the USAID/Malawi Country Development and Cooperation Strategy (CDCS) to improve institutional capacity, increase technology use, and strengthen policies and systems; engage the private sector and civil society organizations to help leverage USAID’s investment; and integrate with other USAID￾funded partners to leverage the existing investments. Evaluation Questions 1. To what extent is the ASPIRE design and its implementation on course to achieve the ASPIRE development objective? 2. How is ASPIRE integrating with other USAID-funded activities and other development partners in Balaka and Machinga? 3. How is the ASPIRE activity coordinating with district government bodies? 4. What gains is ASPIRE achieving through its engagement with the private sector? 5. What are the most significant accomplishments, best practices, and lessons learned from the ASPIRE activity? 6. How does ASPIRE need to adapt its approach to achieve its objectives? February 2018 | ASPIRE Performance Evaluation Report vi DESIGN, METHODS, AND LIMITATIONS This evaluation was designed as an accountability and learning tool for USAID and ASPIRE stakeholders, integrating quantitative and qualitative methodologies within a utilization-focused approach that engaged key stakeholders throughout the process. The mixed-methods design provides breadth (via quantitative data) and depth (via qualitative data) to answer the evaluation questions. Primary source quantitative and qualitative data were collected in June and July 2017, and are supported with activity monitoring data. Quantitative tools comprised a six-task reading assessment; a knowledge, attitudes, and practices (KAP) survey questionnaire for learners; a structured head teacher questionnaire; and a school-based checklist. These tools were based on ASPIRE’s baseline. Qualitative methods drew on appreciative techniques, using key informant interviews and focus group discussions to gain a holistic understanding of stakeholders’ experiences. Sampling used a two-stage design, with a stratified random sampling procedure for quantitative data and purposive sampling for qualitative data. Quantitative analysis involved descriptive and inferential statistics, giving attention to group distributions to identify differences between key subpopulations, particularly between males and females. Qualitative analysis used coding to identify themes. A collaborative synthesis process produced findings by triangulating data across sources. This evaluation has three primary limitations. First, ASPIRE’s internal baseline data were not suitable for comparison with the 2017 performance evaluation data; consequently this evaluation cannot quantify change over time. Second, care is required in interpreting administrative data underlying many of the ASPIRE indicators due to data accuracy and timeliness in the Education Management Information System. Consideration is also required in interpreting results from the 2017 reading assessment, which was based on a previous tool for which piloting results were not documented. Third, only statistics that report confidence intervals or significance tests are generalizable to the population, and the evaluation cannot establish statistical causality. Qualitative data are, by design, not intended to be generalizable to the population. FINDINGS Evaluation Question 1: ASPIRE’s Progress – Qualitative data indicate perceptions of progress toward increasing attendance, progression, and retention, and decreasing dropout rates, but monitoring data were less conclusive. Respondents attributed this progress to the work ASPIRE is doing, and indicated that the cross-sectoral design is a relevant response to the challenge. The holistic approach has yielded benefits as successes under one output, which are seen to contribute to successes under other outputs. Overall, ASPIRE’s implementation is on track to meet implementation targets. Progress toward Output 1 (Improving Reading Skills): 2017 reading assessment data show that upper primary students exhibit the mechanical ability to read fluently in Chichewa and English, but the higher-level competency of reading with comprehension is lacking in both languages. Boys and girls exhibit similar reading fluency, but boys significantly outperform girls in reading comprehension. February 2018 | ASPIRE Performance Evaluation Report vii Weak oral English skills could partially explain the weakness in reading comprehension. Respondents feel that ASPIRE’s inputs are helping improve schools’ educational capacity; ASPIRE’s teacher training and extracurricular activities were singled out as important contributors to success. Progress toward Output 2 (Improving Sexual and Healthcare-seeking Behaviors): 2017 KAP data show that basic HIV/AIDS knowledge among primary and secondary students is strong, but there are gaps in knowledge of certain modes of transmission. Students, teachers, and community groups reported increased access to sexual and reproductive health services for students, largely due to ASPIRE’s work in referrals and school health days, which involve HIV counseling and testing. While overall knowledge of contraceptives was high, actual use among sexually active students was more limited. Sexually active students—both primary and secondary—report relatively low use of condoms. Progress toward Output 3: Reducing Structural and Cultural Barriers: ASPIRE has made substantial progress in reducing structural barriers to girls’ education, particularly in regard to hygiene and sanitation. Respondents felt ASPIRE has reduced cultural barriers to girls’ education and health through policy advocacy at the national level, and at the local level, capacity building and operationalizing strategies that safeguard girls. Students reported that next to teachers, families are an important positive influence on their health and education practices, although many also noted negative influences from family. Respondents mentioned ASPIRE’s work with families infrequently. Evaluation Question 2: Integration with Other USAID-Funded Activities – ASPIRE is integrating well with numerous other USAID-funded activities, particularly around school health days, and sexual and reproductive health service provision through the referral system. Shared objectives and complementary activities with these partners support integration, as do USAID’s joint planning meetings and assistance in coordination. While recognizing progress, implementers identified communication as the primary challenge to integration of USAID partners’ activities; to a lesser extent, implementers pointed to competition for beneficiaries as a challenge. Evaluation Question 3: Coordination with District Government – Government officials reported that ASPIRE is good at coordinating with district offices by ensuring clear communication, facilitating training, and supporting supervision. Community stakeholders saw this collaboration through an increased presence of government officials in schools and communities. Respondents felt that ASPIRE’s coordination with districts could be strengthened through increased monitoring, communication and joint planning, and capacity-strengthening support to district structures. Evaluation Question 4: Engagement with Private Sector – ASPIRE is on track to achieve or exceed its targets for private sector engagement, mostly through engagement with media partners. Perceptions regarding the possibility for further private sector engagement were mixed. Evaluation Question 5: Accomplishments, Best Practices, and Lessons Learned – Respondents singled out ASPIRE’s hygiene and sanitation activities as a key accomplishment, pointing to the construction of changing rooms and distribution of sanitary pads, and improved hygiene that resulted from these. ASPIRE’s community engagement to influence attitudes and encourage collaboration among all stakeholders has led to increased knowledge of health, more support for girls February 2018 | ASPIRE Performance Evaluation Report viii and boys to attend school, and ultimately, greater demand for girls’ health and education. Training was another key accomplishment and best practice respondents highlighted; those receiving training felt this capacity building, particularly for teachers and mothers’ groups, was highly important. Respondents stated that teachers and community actors trained by ASPIRE, particularly mothers’ groups, are a key influence on students’ knowledge of sexual and reproductive health, helping raise their awareness. Finally, successful integration with other USAID partners was reported as a key ASPIRE accomplishment, but it has not been easy; this has led to several lessons learned. Evaluation Question 6: Beneficiaries’ Recommendations on Adaptations – While respondents across stakeholder groups praised ASPIRE’s cross-sectoral design in terms of its holistic approach, they also pointed to substantial implementation management challenges this model creates. Stakeholders at all levels recommended improving monitoring and want better harmonization among USAID partners, and between partners and local-level actors; they recommended making better use of district structures and leveraging partner resources to better integrate activities. Project-level actors pointed to the need to streamline USAID management across the sectors. All respondents wanted continued engagement with the community through awareness-raising campaigns, engaging chiefs to address bylaws, working with parents, motivating local volunteers, and supporting and training mothers’ groups. There is overwhelming support for ASPIRE’s training for teachers, government officials, and community members, with an emphasis on involving more participants and conducting refresher training for those who have already participated. Local stakeholders and students want additional direct contributions from ASPIRE (e.g., toilets, changing rooms, books, and bursaries), but are concerned about sustainability. Local stakeholders recommended that ASPIRE include boys, particularly in the provision of bursaries, to reduce competition and draw more support for girls’ education. Students wanted more extracurricular activities, especially those focused on sexual and reproductive health, as well as openness from parents and teachers regarding these issues. School staff recommended more role model programs. CONCLUSIONS AND RECOMMENDATIONS The evaluation team synthesized the findings with existing research to develop two overarching conclusions. The recommendations are based on input received from ASPIRE, USAID, and activity stakeholders during a workshop with 35 participants, along with evaluation findings, documented best practices, and the perspectives of subject matter experts on the evaluation team. ASPIRE’s cross-sectoral model is a strong program design, reflecting a holistic view of beneficiaries that resonates with all stakeholders. Although managing multiple funding streams in implementation is difficult for managers at all levels, the evaluation findings suggest that ASPIRE’s interventions have the potential to achieve its objectives and offer preliminary evidence supporting the development hypothesis. ASPIRE’s value is greater than the sum of the successes across outputs. The teaching and learning materials, extracurricular activities, and continuing professional development build teachers’ capacity for high-quality literacy instruction and supporting students to February 2018 | ASPIRE Performance Evaluation Report ix develop an appreciation for reading. ASPIRE’s mix of school and community activities promote learners’ knowledge of HIV/AIDS and sexual and reproductive health, and the activity is effectively addressing structural and cultural barriers to girls’ education. At the same time, the holistic design has led to management challenges given the reporting and coordination burdens associated with multiple funding streams and sectors, geographic expansion midway through implementation, and the influx of Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe (DREAMS) initiative funding. The ability of ASPIRE and USAID staff to respond accordingly and maintain fidelity of the cross-sectoral model in the face of these challenges is a notable achievement. These challenges offer a plausible explanation for some of the weaknesses noted by ASPIRE’s management, particularly around activity monitoring. ASPIRE has made strong progress toward implementation targets across all three areas of the results framework, as well as the strategic goal. However, the current pace of change in outputs and the strategic development objective is unclear due to baseline limitations, making it difficult to determine whether ASPIRE will meet these targets in its performance period. Meanwhile, students’ health knowledge and reading fluency are yet to translate into positive health-seeking behaviors and reading comprehension. Findings suggest that gaps remain with regard to literacy and health outcomes, structural needs and cultural barriers to girls’ education, and sustainability. Although students seem to have learned the mechanics of reading in Chichewa and English, they need further support to build the higher order skills of reading with comprehension. In parallel, health results show strong knowledge and positive attitudes related to the Life Skills Curriculum, but this has yet to translate into safe practices that protect girls from unwanted pregnancy and all students from HIV; this gap may partly be linked to limitations in the curriculum. Community- and school-level respondents strongly emphasized the importance of water, sanitation, and hygiene (WASH) infrastructure, and activities that support sanitation and menstrual hygiene management, but ASPIRE has only been able to scratch the surface of structural needs, given available resources. Results indicate that ASPIRE has offered a strong response to the cultural barriers to girls’ education. Next steps include garnering families’ and communities’ full support for girls’ health and education needs. Finally, respondents were concerned about ASPIRE’s sustainability, particularly components focused on providing physical or monetary support, such as school bursaries supported through DREAMS funding, sanitation facility construction through WASH funding, and seed funding to mothers’ groups. Respondents worried that ASPIRE’s successes might be threatened when this funding ends. Recommendations for ASPIRE  Continue implementing high-quality interventions that reflect the cross-sectoral strengths ASPIRE brings. If training remains a focus, an increase in monitoring training quality and outcomes should be included in ASPIRE’s activities. Further improving coordination with district government bodies, such as the District Education Network, could help grow the dividends of the cross-sectoral model. February 2018 | ASPIRE Performance Evaluation Report x  Continue to support community-level actors who work across output areas. However, ensure that ASPIRE messaging is not unintentionally translated into unanticipated negative outcomes.  Strengthen indicator monitoring systems that feed into reporting with a goal to improve data quality and report clarity. Beyond improving data quality for indicator reporting, ASPIRE should ensure that monitoring systems are designed to assess the quality of activities on an ongoing basis and provide up-to-date feedback to stakeholders. Quality monitoring is particularly important for training programs, which could benefit from routine collection of participant feedback and post-training follow-up.  Introduce or modify activities that can capitalize on the existing strong knowledge and attitudes to help translate these into positive practices for both literacy and health. In education, classroom and extracurricular instruction needs practical models to help translate fluent reading into reading comprehension, including explicit instruction of diverse comprehension strategies. Explore how these strategies can target girls’ comprehension to help close the gaps identified in this evaluation. In health, interventions should target ways to help students turn knowledge into practice. This may be challenging within schools, given the constraints of the existing Life Skills Curriculum. Consequently, it will be important to consider ways to disseminate these messages through community-based programming and ensure that policymakers understand the limitations of the current curriculum.  Because ASPIRE has provided a number of direct inputs in the first 2.5 years, outline and implement models for sustainability in the final year to ensure that district, school, and community structures can sustain interventions once direct financing and support ends.  Use internal monitoring, evaluation, and learning activities to help answer new questions raised by this evaluation. Two particular questions emerge as important: (1) What factors are driving differences in girls’ and boys’ oral and reading comprehension? and (2) What interventions hold the greatest potential to translate HIV/AIDS and sexual and reproductive health knowledge into positive practices and healthcare-seeking behaviors? Recommendations for USAID/Malawi  Continue investing in and supporting implementation of a holistic approach to girls’ empowerment activities that reflect understanding of beneficiaries’ needs and context. The cross-sectoral activity design reflected in ASPIRE is a model for future activities in this area. USAID/Malawi offices should continue to seek opportunities to support activities, drawing from multiple funding streams where these links are logical.  Strengthen internal management procedures and external communication for activities with multiple funding streams to ensure that offices agree on priorities and USAID speaks with one voice in articulating these priorities. Activities with multiple funding streams could benefit from a champion at USAID who views them as a whole, rather than just the results their office supports. Offices should ensure that they are mutually supportive in their February 2018 | ASPIRE Performance Evaluation Report xi communication to implementing partners and view the project from the perspective of its ultimate intended goal, not just sectoral targets.  Support expansion of successful ASPIRE interventions to other districts where resources are available and prioritize where investments are made. Work with the Government of Malawi to determine its role in scale-up and align USAID programming with these efforts.  Approach activity monitoring and evaluation from a holistic perspective, considering the total reporting burden for the implementation team. Minimizing indicator revisions improves consistency of data, and enables better tracking and comparison of results over time. Approaching evaluation needs for the entire lifecycle at the outset of an activity can lead to more robust evaluations. For example, limitations in baseline studies affect the strength of future evaluations; for future activities, USAID should ensure that baselines are of sufficient quality and documentation to allow effective comparison by future external evaluation teams. Planning should begin now for ASPIRE’s endline evaluation. Explore learning activities that can help answer the additional questions raised by this evaluation.  Support ASPIRE’s advocacy work, particularly around the Life Skills Curriculum, by reinforcing these messages in communication with the Government of Malawi. Ensuring that policymakers understand the constraints of the current curriculum could help produce openings for curricular revisions. Recommendations for Central Government of Malawi  Use learning from ASPIRE to make decisions on policy, curricula, and scale-up. Continue supporting ASPIRE initiatives, particularly around operationalizing policy and curricula.  Scale up promising ASPIRE interventions to achieve national reach. In particular, expand use of the Life Skills Curriculum and the associated continuing professional development, school health days and the school-based referral system, and school WASH committees, which can help achieve the government scale-up of improving WASH infrastructure in schools.  In conjunction with scale-up, work with ASPIRE to create a sustainability plan during the last year of implementation aimed at ensuring institutional arrangements for uptake. Identify the ministry responsible for supporting ASPIRE’s community engagement work to ensure that work continues with WASH committees, traditional leaders, and mothers’ groups.  Encourage cross-sectoral engagement at the national level, particularly in aligning policies across departments. For example, consider revision of the National Education Policy, which limits sexual and reproductive health education on school grounds.  Convene stakeholders from all the relevant ministries and departments, along with subject matter experts, to comprehensively review and update the Life Skills Curriculum. This evaluation offers insights into strengths and limitations of the current curriculum that can serve as a starting point to identify potential revisions. February 2018 | ASPIRE Performance Evaluation Report xii  Under the leadership of the Ministry of Education, Science and Technology, review and update the upper primary reading curriculum to align it with the National Reading Program used in lower primary. These revisions should include emphasis on multiple comprehension strategies to help students learn to understand both written and oral language. Develop corresponding benchmarks for Chichewa and English reading skills; this evaluation provides some data that can be used in initiating benchmark conversations. Recommendations for District-Level Government Offices  Continue to host coordination meetings at District Executive Committee to encourage collaboration among implementing partners.  During the last year of implementation, work with ASPIRE to create a sustainability plan aimed at taking over some of the successful activities ASPIRE has been implementing.  Provide feedback to central Government of Malawi ministries that are supporting promulgation of successful ASPIRE activities to ensure that policies, guidance, and support are relevant and user-friendly. February 2018 | ASPIRE Performance Evaluation Report 1 1. EVALUATION PURPOSE AND QUESTIONS USAID awarded EnCompass LLC a task order to implement the performance evaluation of the Malawi Girls’ Empowerment through Education and Health Activity (ASPIRE) on May 1, 2017. This external performance evaluation, conducted 2.5 years after ASPIRE began, establishes the activity’s progress against its objectives, proposes adaptation and remedial measures for ASPIRE’s implementation, and captures lessons learned for application in future girls’ empowerment, health, and education programming in Malawi. The performance evaluation achieves these goals by responding to six evaluation questions (Exhibit 1). Question 1 is answered at the level of the three activity outputs and the development objective to establish the added value of the cross-sectoral approach beyond progress against each objective (see Section 2.2, ASPIRE Design and Theory of Change). The evaluation questions are taken from the scope of work and were reviewed by the evaluation team, USAID/Malawi, ASPIRE staff, and activity stakeholders during the inception phase (May 1 through 19, 2017). This review confirmed that the questions were relevant to stakeholders’ needs and did not require modification. The inception phase culminated with the Inception Report, which documents this stakeholder engagement process and the evaluation design used to meet the requirements of the scope of work. Annex 1 presents the full evaluation scope of work. USAID approved the final Inception Report on June 19. The primary intended audience for the evaluation is USAID/Malawi, specifically its Education and Health, Population, and Nutrition offices, including the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) team. ASPIRE implementing partners and the implementing offices of the Government of Malawi are additionally expected to benefit from the evaluation results. During the inception phase, USAID/Malawi, the ASPIRE implementation team, and partners from the Ministry of Education, Science and Technology (MoEST) expressed their intention to use the Exhibit 1: Evaluation questions 1. To what extent are the ASPIRE design and its implementation on course to achieve the ASPIRE development objective? 2. How is ASPIRE integrating with other USAID￾funded activities and other development partners in Balaka and Machinga? 3. How is ASPIRE coordinating with district government bodies? 4. What gains is ASPIRE achieving through its engagement with the private sector? 5. What are the most significant accomplishments, best practices, and lessons learned from ASPIRE? 6. How does ASPIRE need to adapt its approach to achieve its objectives? February 2018 | ASPIRE Performance Evaluation Report 2 answers to the evaluation questions (the evaluation results) to inform implementation in ASPIRE’s final year. USAID/Malawi, as well as the broader Agency and community of practice, might also use the results to inform the design of future activities aimed at girls’ empowerment. This utilization￾focused purpose guided the evaluation through an inclusive process for generating recommendations during review of the draft report and the findings workshop that EnCompass facilitated at the end of the evaluation cycle. February 2018 | ASPIRE Performance Evaluation Report 3 2. BACKGROUND 2.1 CONTEXT Despite recent improvements in access, education in Malawi is characterized by low academic achievement and poor retention rates for boys and girls. Overall, only 50 percent of students complete primary school, and of those, only 68 percent pass the primary school leaving exam (34 percent of the total enrolled in Standard 1). National education data indicate that the situation is even direr for girls, of whom only 29 percent who enroll in Standard 1 progress to Standard 8, and of those, only 37 percent transition to secondary school (in other words, only 11 percent of those who enroll in Standard 1 transition to secondary school). These high repetition and dropout rates discourage parents from sending children, especially girls, to school, fueling a vicious cycle. Learning assessments repeatedly show that Malawian students are failing to acquire basic literacy skills—a key achievement factor influencing school completion, particularly in primary school because learning to read affects students’ ability to “read to learn” (USAID/Malawi Country Development Cooperation Strategy [CDCS] 2013–2018). The barriers to girls’ achievement in upper primary and secondary schools are complex and include social, economic, and cultural factors that influence norms and behaviors at individual, community, and societal levels. These factors are reinforced by gender-inequitable attitudes held by teachers, parents, and the wider community. Although the Government of Malawi has committed to challenging these barriers, progress has been slow. Malawi continues to have one of the world’s highest rates of child marriage, with approximately one in two girls married by the age of 18. A key challenge to ending child marriage in Malawi is entrenched attitudes that accept the practice. Child marriage is also closely linked to poverty, with girls in rural areas often married very young to improve a family’s financial status (United Nations Children’s Fund [UNICEF] 2016). Adolescent girls and young women in Malawi also face staggering rates of gender-based violence, with 68.4 percent of 18- to 24-year-olds and 76.3 percent of 13- to 17-year-olds reporting multiple incidents of sexual abuse before the age of 18. Studies show that girls in Malawi experience gender￾based violence not only on school grounds, perpetrated by fellow classmates and teachers, but also while traveling to and from school (Mellish et al. 2015). One household study, which looked at the effects of gender-based violence on girls’ education, found that 60.9 percent of girls experiencing gender-based violence reported that their experience had resulted in performance problems at school, with a small percentage (3.3 percent) reporting that they had stopped going to school as a result (Bisika et al. 2009; PEPFAR 2015). In the same study, 3.8 percent of girls who reported being inappropriately touched also reported that the perpetrator was a teacher. Finally, Malawian girls face substantial barriers to their health. In the most recent Demographic and Health Survey (DHS), Malawi’s maternal mortality rate was 439 maternal deaths per 100,000 live births (DHS 2015–2016), the 24th highest in the world. The Malawi Population-Based HIV February 2018 | ASPIRE Performance Evaluation Report 4 Impact Assessment—a household-based national survey conducted between November 2015 and August 2016 to measure the status of the national HIV response (Malawi Ministry of Health and Columbia University 2016)—showed a 0.37 percent incidence of HIV among adults aged 15 to 64 in Malawi: 0.48 percent among females and 0.25 percent among males. This corresponds to approximately 28,000 new cases of HIV each year among adults in Malawi. It also shows that the disparity in HIV prevalence by sex is the most pronounced among young adults: among 25- to 29- year-olds, it is three times higher among females (14.1 percent) than males (4.8 percent). 2.2 ASPIRE DESIGN AND THEORY OF CHANGE ASPIRE is designed to improve girls’ achievement—and ultimately, empowerment—in upper primary and secondary school by responding to the aforementioned challenges. In so doing, ASPIRE supports the USAID/Malawi CDCS goal of improved quality of life for Malawians by contributing to Development Objective 1, “Social development improved.” ASPIRE began in December 2014 with $10 million in funding, working in Balaka and Machinga districts. In April 2016, USAID added $5.3 million in funding from the Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe (DREAMS) initiative to support new ASPIRE activities in Machinga and Zomba districts. At the same time, ASPIRE expanded its existing interventions to Zomba district. Consequently, Machinga has received the full intervention since 2014, Zomba has received the full intervention since 2016, and Balaka has received the intervention minus the DREAMS components since 2014. Exhibit 2 describes ASPIRE’s funding streams and Exhibit 3 illustrates its interventions by district. Exhibit 2: ASPIRE funding streams ASPIRE is a 4-year, $16.2 million USAID activity designed to align with the Malawi CDCS emphasis on cross￾sectoral integration, programmatically and financially. Four intergovernmental initiatives and USAID departments fund ASPIRE:  Basic Education: $7 million  PEPFAR Orphans and Vulnerable Children: $2 million  DREAMS: $5.3 million (2016-2018)  Water, Sanitation, and Hygiene (WASH) portfolio: $1.9 million USAID funded the activity under Cooperative Agreement No. AID-612- A-15-00001, December 17, 2014 through December 16, 2018. February 2018 | ASPIRE Performance Evaluation Report 5 Exhibit 3: ASPIRE interventions by district ASPIRE is implemented by a consortium of partners, led by Save the Children, with support from the Forum for African Women Educationalists (FAWEMA), the Creative Centre for Community Mobilisation (CRECCOM), and the Malawi Institute of Education (MIE). Building on the evaluation scope of work, ASPIRE’s annual work plan, and the Activity Monitoring, Evaluation, and Learning Plan, the ASPIRE theory of change posits three essential elements that must occur across a variety of systems and actors to achieve sustainable impact: (1) IF evidence-based approaches to develop girls’ foundational literacy skills and a gender-sensitive learning environment—inclusive of WASH—lead to improved learning outcomes; and (2) IF collective community dialogue and action positively change attitudes and behaviors, directly affecting the causes of girls’ dropouts to increase retention; and (3) IF girls are empowered with the knowledge, motivation, and skills to be effective agents in their personal development, THEN girls’ achievement in upper primary and secondary school will improve. This theory of change emphasizes that girls are actors in their personal development, and their development cannot be considered outside the social web of relationships, influential actors, and societal norms and structures that influence them. The changes are accomplished through activities aimed at three results:  Output 1: Reading skills for girls in upper primary school improved  Output 2: Adoption of positive sexual and healthcare-seeking behaviors among youth ages 10–19 increased  Output 3: Key structural and cultural barriers for girls ages 10–19 decreased. The ASPIRE results framework, presented in Exhibit 4, presents the pathways through which these outputs contribute to the development objective, in the framework of the development hypothesis and the theory of change. February 2018 | ASPIRE Performance Evaluation Report 6 Exhibit 4: ASPIRE results framework 2.3 ASPIRE INTERVENTIONS At present, ASPIRE works to benefit girl learners in all primary and secondary schools in Balaka, Machinga, and Zomba districts. According to 2016 Education Management Information System (EMIS) data collected from District Education Management (DEM) offices, there are 538 primary schools and 82 secondary schools across the three districts. Output 1 Reading skills for girls in upper primary school improved 1.1: Reading systems for students, particularly girls, in Standards 4–8 improved 1.2: Greater engagement, accountability, and transparency by communities, the private sector, and the public to support reading 1.3: Reading instruction in Standards 4–8 improved, with emphasis on increased access to gender-sensitive teaching and learning materials, especially for girls Output 2 Adoption of positive sexual and healthcare-seeking behaviors among youth 10–19 increased 2.1: Capacity of teachers and district trainers to provide comprehensive sexual and reproductive health education to youth ages 10–19 increased 2.2: Risk mitigation and pregnancy prevention behaviors adopted 2.3: Provision for HIV-positive students delivered 2.4 [DREAMS]: Core HIV and violence prevention package for adolescent girls (ages 10– 19) provided to increase their knowledge, self-efficacy, and risk reduction Output 3 Key structural and cultural barriers for girls ages 10–19 decreased 3.1: Engagement and accountability by communities to support improved educational outcomes for girls and boys increased 3.2: Capacity of community groups to support and advocate for girls’ empowerment to stay in and complete school increased 3.3: Utilization of successful and sustainable performance￾based financing mechanisms to support completion of secondary school increased 3.4: Access to adequate water, sanitation, and hygiene (including menstrual hygiene management commodities) improved 3.5: School-based strategies, guidelines, and/or policies to safeguard the girl child adopted or operationalized 3.6 [DREAMS]: Provide safe school environments ASPIRE Strategic Objective: Improved achievement of girls in upper primary and secondary school Reducing teenage pregnancy in Malawi by increasing use of key sexual and reproductive health practices and services, reducing girls’ school dropout rate, and increasing re-entry rate after pregnancy February 2018 | ASPIRE Performance Evaluation Report 7 In its first 2.5 years, ASPIRE has conducted numerous activities designed to achieve outputs 1, 2, and 3, including teacher training, materials development and dissemination, extracurricular activities, school health days in DREAMS-supported districts, improvements to school water and sanitation facilities (including for menstrual hygiene management), and engagement of mothers’ groups to produce menstrual hygiene management commodities. Throughout these activities, ASPIRE has engaged all actors it interacts with to confront cultural barriers that impede girls’ achievement. In addition to these specific approaches targeting the three outputs, ASPIRE employs strategies aligned with the USAID/Malawi CDCS. Specifically, ASPIRE activities:  Reflect the three CDCS cross-cutting issues of improving institutional capacity, increasing technology use, and strengthening policies and systems by coordinating with Government of Malawi structures  Engage the private sector and civil society organizations to help leverage USAID’s investment  Integrate with other USAID-funded development partners to leverage the existing investments, as well as leverage other donors’ investments in Save the Children Malawi programs, such as the Keeping Girls in School, known as KGIS (funded by UK Aid Direct) and Reducing Teen Pregnancy Project, known as RTP (funded by Norwegian Agency for Development Cooperation). This approach aligns with USAID/Malawi’s “3-C” approach (co-locating, coordinating, and collaborating) and builds synergy with other programs and initiatives funded by other parts of the Agency, such as the DREAMS initiative. The following sections provide additional detail on ASPIRE’s interventions, organized by the three outputs. 2.3.1 IMPROVE READING SKILLS (OUTPUT 1) ASPIRE has facilitated an ongoing continuing professional development program for primary school teachers and managers to improve the quality of reading instruction in the classroom. Teacher training has focused on teachers of Standards 4 through 8, emphasizing instruction for advanced literacy skills. This training complements USAID’s Early Grade Reading Assessment (EGRA) (2013–2016) and Malawi Early Grade Reading Improvement Activity (MERIT) (2015–2020), reflecting the Mission’s 3-C approach. This continuing professional development program additionally includes training on gender-sensitive pedagogy. Although literacy instruction in Malawi is intended to transition from Chichewa to English in Standard 5, early on, an ASPIRE needs assessments revealed that teachers’ own command of English was a substantial barrier to their teaching of English literacy. In response, ASPIRE added the English Language Proficiency Pilot, designed to improve teachers’ English language skills and minimize this barrier. February 2018 | ASPIRE Performance Evaluation Report 8 To complement these activities, ASPIRE disseminates teaching and learning materials to primary schools, for use by teachers and students, and teachers receive a manual as part of the continuing professional development program. Additionally, ASPIRE has supported schools to conduct fairs that promote positive attitudes toward the importance of reading. 2.3.2 ADOPT POSITIVE SEXUAL AND HEALTHCARE-SEEKING BEHAVIORS (OUTPUT 2) One of ASPIRE’s key interventions to improve adolescents’ sexual and healthcare-seeking behaviors is to support the adoption of the MoEST’s Life Skills Curriculum. This curriculum existed before ASPIRE, but widespread discomfort limited implementation at the school level; in particular, previous assessments documented teachers’ challenges in delivering sensitive aspects of the curriculum. This challenge was compounded by a shortage of the Life Skills Curriculum textbook. To address these impediments, ASPIRE worked with MoEST to distribute the textbook to schools, and it provided activity cards designed to facilitate discussion of sexual and reproductive health topics in classroom and extracurricular settings. ASPIRE paired these materials with a continuing professional development program and a manual for teachers to support them in delivering these topics, with a focus on building teachers’ comfort. ASPIRE finalized the manual in early 2016. In ASPIRE’s original design, this work targeted secondary schools only; it was expanded to primary schools with the addition of DREAMS funding in 2016. DREAMS funding also supports training on the Teachers’ Code of Conduct, which promotes discipline and professionalism among teachers in order to create safe spaces for students in school and, ultimately, improve retention of girls. To improve sexual and reproductive health service delivery by providing students with easier access to services—including HIV testing and counseling—with the introduction of DREAMS funding, ASPIRE began facilitating semiannual school health days. Traditionally, there has been hesitancy to conduct HIV testing at schools due education policies, which do not allow for distribution of family planning information or resources (often done as part of HIV counseling) on school grounds. To support access while adhering to MoEST policy, ASPIRE builds partnerships between schools and service delivery organizations to provide these services near, but not on, school grounds during the school health days; each school in Machinga and Zomba districts (the DREAMS focus area) holds 2 health days per academic year. ASPIRE coordinates with service providers from other USAID projects, as well as the government healthcare system—for example, with Banja la Mtsogolo (BLM/Marie Stopes) in Zomba and Population Services International (PSI) in Machinga. Starting in October 2016, all DREAMS partners have participated in piloting a common referral system and associated tools to facilitate students’ access to health services. As a DREAMS partner, ASPIRE supports the referral system at the school and community level, ensuring teachers, mothers’ groups, and other community actors use the system. ASPIRE also placed boxes in schools to improve tracking of student referrals. School health days and referrals complement teacher training, which emphasizes the link between the education and health sectors as a way of promoting health-seeking behaviors. More broadly, February 2018 | ASPIRE Performance Evaluation Report 9 ASPIRE has held events with district-level government officials, facilitated a girls’ education network, and hosted policy advocacy events with the Government of Malawi. 2.3.3 DECREASE STRUCTURAL AND CULTURAL BARRIERS (OUTPUT 3) To reduce structural barriers to girls’ education, a key part of ASPIRE’s work has been to improve school WASH. In 2015, ASPIRE conducted a needs assessment and developed a latrine design targeted at increasing accessibility for girls and students with special needs, two groups facing high access barriers. The menstrual hygiene management facilities and latrine construction began in 2016, targeting the highest-need schools identified during the needs assessment; USAID provided funding for construction of these facilities at 49 schools, spread across the three ASPIRE districts. Additionally, ASPIRE supports all schools with hand-washing stations and water treatment products to ensure safe drinking water. ASPIRE has also facilitated community-level capacity building across the entire intervention area aimed at reducing cultural barriers to girls’ education. Training has targeted mothers’ groups, parent￾teacher associations (PTAs), school management committees, and traditional leaders, working to influence attitudes toward the importance of girls’ completing school, prevention of childhood marriage, and factors that affect girls’ academic achievement. These community groups identify female role models and male champions who reinforce the core ASPIRE messages. Trainings also promote the importance of menstrual hygiene management facilities at schools, encouraging communities to address these structural barriers in schools where ASPIRE is not able to directly support infrastructure improvements. Based on the ASPIRE training, mothers’ groups form auntie/agogo clubs that teach girls about sexual health and the importance of staying in school, produce menstrual hygiene management commodities for school-going girls, and form village savings and loan programs to generate resources that can be used to support girls’ hygiene commodities and infrastructure. Finally, DREAMS funding has supported school block grants in Machinga and Zomba districts, which ASPIRE provided to all 82 secondary schools and 125 primary schools. These block grants support initiatives selected by schools as having the highest potential for promoting girls’ education, and have been used for hygiene facility construction and scholarships for school fees. February 2018 | ASPIRE Performance Evaluation Report 10 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS This external performance evaluation, designed as an accountability and learning tool for USAID/Malawi and ASPIRE activity stakeholders, integrated quantitative and qualitative methodologies within a utilization-focused approach that engaged key stakeholders in evaluation planning and results validation. This section summarizes the final achieved evaluation design; see Annex 2 for full details. 3.1 EVALUATION DESIGN This performance evaluation used a mixed-methods design that provides both breadth (via quantitative data) and depth (via qualitative data) to answer the evaluation questions. This design offers the flexibility to not only capture what achievements are occurring at a generalizable level, but also gain a deeper understanding of why change is or is not happening and how achievements are being made. Primary-source quantitative and qualitative data were supported with monitoring data and implementation information obtained through a document review. Together, these data types better informed results and led to more nuanced recommendations by balancing a practical assessment of progress toward ASPIRE indicators, with consideration of the complex social dynamics that the development hypothesis attempts to influence and the sustainability of the interventions. Quantitative data were collected through a school-based survey in June and July 2017, which was designed to produce generalizable data that can be inferred to the intervention area population. These primary-source quantitative data reflect a single cross-section—ASPIRE beneficiaries at the time of the evaluation. As a short performance evaluation, occurring midway in the activity’s implementation, a design featuring a counterfactual was not possible. Additionally, statistical estimation of change over time was not possible due to baseline limitations. The limitations of the quantitative data’s single cross-sectional design are discussed in Section 3.5. EnCompass collected qualitative data from a purposive sample targeting schools and their surrounding communities. This design enabled triangulation of results at the local level during data collection, reflecting the ASPIRE development hypothesis and theory of change, which hold that sociocultural structures outside the school are key enablers of and barriers to girls’ empowerment. The purposive sample was designed to capture the breadth of stakeholder perspectives, recognizing that idiosyncratic factors might have strong differentiating effects on beneficiaries’ experiences and ASPIRE’s results. Section 3.3.2 provides detail on purposive sampling categories. February 2018 | ASPIRE Performance Evaluation Report 11 3.2 DATA COLLECTION TOOLS AND METHODS Data collection methods are described briefly below; see Annex 2 for details and Annex 8 for the complete set of tools.  Document Review: The evaluation team developed a document review matrix and accompanying guidelines to extract information in a structured approach. The document review collected activity monitoring data and provided information on background, context, and ASPIRE activities, which the evaluation team triangulated with other data sources. The desk review began prior to inception and continued through the conclusion of primary data analysis, as new documents and data became available. An evaluation specialist at EnCompass’ home office supervised the desk review.  Quantitative Tools and Methods: The evaluation team used four quantitative tools adapted from ASPIRE’s baseline: (1) a six-task reading assessment to measure literacy skills (Chichewa and English) of learners in Standards 5 and 6; (2) a knowledge, attitudes, and practices (KAP) survey questionnaire to assess sexual and healthcare-seeking behaviors, and determinants among learners in Standards 5 and 6 and Forms 1 and 2; (3) a structured head teacher questionnaire to collect school-level information on ASPIRE activities and results across all three outputs; and (4) a structured observation school checklist to document school materials and infrastructure.  Qualitative Methods: The evaluation team used two qualitative methods, drawing on appreciative evaluation techniques to elicit successes: (1) key informant interviews to allow deep exploration and build “strong” narratives that provide a holistic understanding of stakeholders’ experiences with ASPIRE; and (2) focus group discussions to spur discussion and build collective narratives around the evaluation questions, and gain insight into beneficiaries’ different perceptions. Interviews and discussions took place in a mix of Chichewa and English, depending on participants’ preferences, and collected information across all three ASPIRE outputs and the development objective. Each method used a semi￾structured guide that enabled probing. Section 3.3.2 provides respondent categories. All data collectors participated in 5 days of training on quantitative tools and 3 days of training on qualitative tools, emphasizing quality-control procedures and evaluation ethics, with extensive opportunities for practice and a real-world pilot. Training for the reading assessment included a simulation of tool administration under controlled conditions to promote inter-rater reliability. All quantitative tools were administered on tablets, and qualitative data were verified using audio recording. For details on evaluation ethics, see Annex 2. February 2018 | ASPIRE Performance Evaluation Report 12 3.3 SAMPLING DESIGN 3.3.1 QUANTITATIVE SAMPLE DESIGN AND WEIGHTING Quantitative data collection relied on a two-stage stratified random sampling procedure to select schools and students, with schools serving as clusters. Because ASPIRE interventions target the population of schools in the intervention districts, the sample drew from all registered schools in a given district; there was no separation between intervention and comparison schools. In Stage 1, a random sample of schools was drawn using probability proportional to size, stratified by district. For Stage 2, a simple random sample of learners was drawn in the targeted grades at each selected Stage 1 school, stratified by sex and grade. In primary schools, the target grades were Standards 5 and 6; in secondary schools, the target grades were Forms 1 and 2. Because ASPIRE interventions and expected results differ between primary and secondary schools, separate samples were drawn for each school type and the two groups are treated as separate populations for purposes of analysis. All inferential analysis presented in this report uses probability weights (see formulas in Annex 2). Exhibit 5 in the next section presents the final achieved sample, which exceeded the target. 3.3.2 QUALITATIVE SAMPLE DESIGN The qualitative sample reflects a purposive two-stage design to capture the breadth of school types and stakeholders with the potential to influence ASPIRE’s results. At the first stage, 20 schools were selected from the three districts. At the second stage, interview and focus group respondents were selected from within schools and non-school–based actors were selected from schools’ catchment areas. Data were also collected from central- and district-level respondents. The target qualitative sample was designed to approximate saturation; the final sample, with more than 100 interviews and focus groups, exceeded the target qualitative sample and is presented in Exhibit 6 in the next section. The qualitative school sample was drawn from the final quantitative Stage 1 sample. Purposive sampling ensured that the qualitative sample included schools from each district, grade level, school setting (rural or urban), and both high- and low-performing schools. Because quantitative data collection could not be completed before qualitative data collection, as originally intended, ASPIRE staff assisted in classifying schools as high- or low-performing. This purposive category facilitated identifying best practices and significant accomplishments (Evaluation Question 5), implementation challenges (Evaluation Question 6), and barriers to and enablers of success. A stakeholder analysis during inception identified respondent categories for the second stage of qualitative sampling. Categories represent each level of the ASPIRE theory of change: students (Standards 5 and 6 and Forms 1 and 2), school staff (teachers, head teachers, and teacher trainers), community groups (PTAs, WASH coordinators and committees, Youth-Friendly Health Services providers, School Management Committees, Life Skills patrons, HIV coordinators and initiation counselors), mothers’ groups, government officials (central and district levels, including district education managers, coordinating primary education advisors, primary education advisors, and senior education methods advisors), nongovernmental organizations (NGOs) and development February 2018 | ASPIRE Performance Evaluation Report 13 partners (ASPIRE implementing partners working as subcontractors to Save the Children and USAID and DREAMS activities with which ASPIRE coordinates), private sector partners, and project-level actors (ASPIRE and USAID/Malawi staff). The evaluation team distributed these categories proportionally across schools to ensure that the qualitative sample included a suitable number of respondents from each category. 3.4 ANALYSIS The evaluation team analyzed quantitative data in Stata version 14, first using descriptive statistics, and then proceeding to inferential techniques. Analysis gave particular attention to group distributions to identify differences between key subpopulations, and disaggregated all data by sex to compare similarities and differences between males and females. For continuous variables from the reading assessment, the team examined all distributions for normality to inform selection of the correct measures of central tendency (mean versus median) and conducted zero score analysis. All differences that are statistically significant at the 5 percent level are noted as such; tests that yielded insignificant differences are similarly noted. Where the text notes neither significance nor insignificance, it means tests were not performed due to unsuitability. Although Annex 3 presents data disaggregated by district and school setting (urban or rural), these disaggregations are calculated using descriptive techniques due to the sampling limitations described in Section 3.5; thus, these results are for informational purposes only and should not be assumed to be reflective of the broader population. In general, the report disaggregates data by sex even where differences are not significant. Unless specifically noted, differences should not be construed as statistically significant. The evaluation team analyzed qualitative data using Dedoose, a cross-platform application that allows collaborative coding by multiple team members and assessment of inter-coder reliability. Qualitative analysis used a two-stage approach to facilitate thematic analysis that addressed each evaluation question and provided insight into the elements underlying ASPIRE’s theory of change. Prior to analysis, the team created and piloted a codebook to ensure relevance of the coding structure to the data and consistent code application by the analysts. The team then refined the codebook and assessed inter-coder reliability by having each team member involved in qualitative analysis code the same document. Code application was compared and discussed to ensure high reliability. This coding structure was applied to all transcripts during the first stage of coding, reflecting a deductive approach to organizing data. The codebook allowed for identification of emerging themes to identify ASPIRE’s contributions to improving achievement in upper primary and secondary schools. In the second stage, the evaluation team analyzed each code to generate emergent themes through an inductive process; inductive analysis avoids presupposing hypotheses regarding respondents’ experiences, thus allowing unexpected results to surface based on respondents’ most salient points. Following analysis of each data type, the evaluation team used a collaborative process to compare emergent themes, triangulate data across sources, and synthesize findings. This approach allowed the evaluation team to view the data from different perspectives and capture learning. February 2018 | ASPIRE Performance Evaluation Report 14 3.5 LIMITATIONS This performance evaluation is limited by the following factors:  Lack of Comparable Baseline: ASPIRE conducted an internal baseline in 2015, before launching activities, but baseline data are not suitable for statistical comparison to 2017 performance evaluation data. As a result, this performance evaluation is limited in that it cannot assess change over time for quantitative indicators. Specific factors limiting the ability to compare baseline and performance evaluation data are presented in Annex 2 (Sampling Design).  Limitations of Data Sources and Tools: This limitation has two components. First, care should be taken in interpreting administrative data from the EMIS and the District Education Management Information System (DEMIS), which comprise many ASPIRE indicators, but are frequently of questionable accuracy and timeliness. Second, the reading assessment tool is based on the ASPIRE baseline tool and EGRA instruments used in other USAID activities. There is no record of piloting and psychometric analysis of item validity for the former instrument, and the latter was not originally intended for Standards 5 and 6. Additionally, a lack of grade-level reading benchmarks for Chichewa and English in the upper primary levels means that reading assessment development was not guided by standards agreed upon by experts. Results should not be compared across the two languages, and consideration should be given when interpreting results as to whether passages are an accurate reflection of grade-level expectations.  Limitations of Sampling Design: This limitation has three components. First, the quantitative sample provides reasonable specificity for estimating health and literacy results among ASPIRE beneficiaries given the evaluation’s purpose; however, as with any evaluation, a larger sample could provide more specific estimates (i.e., narrower confidence intervals) and detect additional differences between subpopulations. Second, quantitative data reflect an observational evaluation design featuring a single cross-section (one point in time); as a result, the evaluation cannot statistically attribute results to ASPIRE. Finally, qualitative data are, by design, not intended to be generalizable to the population. February 2018 | ASPIRE Performance Evaluation Report 15 4. DESCRIPTION OF SAMPLE This section outlines key characteristics of the performance evaluation sample to provide context for the findings. Exhibit 5 presents the overall quantitative sample and Exhibit 6 presents the qualitative sample; sample targets were surpassed for both data types, but no private sector actors responded to interview requests during the data collection period. Exhibit 47 in Annex 2 disaggregates the quantitative student sample by grade. Exhibit 7 and Exhibit 8 provide details on the qualitative sample. The rest of this section presents descriptive statistics for the quantitative sample respondents. Exhibit 5: Quantitative sample overview Learners: sample stage 2 Schools: sample stage 1 Primary Secondary Head teacher questionnaire respondents* District Primary Secondary Male Female Total Male Female Total Male Female Total Balaka 6 4 99 96 195 63 64 127 8 (3) 2 (1) 10 (4) Machinga 6 4 97 95 192 64 64 128 8 (4) 2 10 (4) Zomba 6 4 97 96 193 63 63 126 7 (2) 3 (2) 10 (4) Total 18 12 293 287 580 190 191 381 23 7 30 * Parentheticals denote participants with other titles. Exhibit 6: Qualitative sample by respondent type, sex, and method Key informant interviews* Focus group discussions Respondent category Male Female Mixed Sex Male Female Mixed Sex Total Community group 13 7 (1) 5 (1) 0 0 2 27 Government official 6 2 1 (1) 0 0 0 9 Mothers’ group 0 2 (1) 0 0 8 0 10 NGO/Development partner 4 1 1 (1) 0 0 0 6 Private sector 0 0 0 0 0 0 0 Project-level actor 2 5 1 (1) 0 0 0 8 School staff 10 10 1 (1) 0 0 8 29 Students 0 0 0 10 13 0 23 Total 35 27 (2) 9 (5) 10 21 10 112 Grand total 71 41 * Parentheticals denote number of interviews conducted as group interviews. February 2018 | ASPIRE Performance Evaluation Report 16 Exhibit 7: Qualitative sample by district, school type, and setting Primary Secondary District Rural Urban Semi-urban Rural Urban Semi-urban N/A* Total Central level n/a n/a n/a n/a n/a n/a 11 11 Balaka 24 0 0 3 2 1 4 34 Machinga 21 6 1 5 0 0 7 40 Zomba 2 11 0 8 2 0 4 27 Total 47 17 1 16 4 1 26 112 Grand total 65 21 * N/A includes government officials, project-level actors, and other non-school respondents. Exhibit 8: Total number of group interview and group discussion respondents Group interview participants Group discussion participants Districts Male Female Male Female Total Central level 2 2 0 0 4 Balaka 0 4 46 59 109 Machinga 5 2 50 39 96 Zomba 3 3 24 52 82 Total 10 11 120 150 291 Grand total 21 270 4.1 CHARACTERISTICS OF HEAD TEACHERS IN THE SAMPLE For each of the 30 schools in the quantitative sample, the evaluation team surveyed the school’s head teacher (or the deputy head, when the head teacher was unavailable), yielding a sample of 30 head teachers and deputy head teachers. The respondents had a wide range of experience in their positions (1 year to 28 years), as Exhibit 9 summarizes. Three-fifths of the respondents had a Malawi School Certificate of Education (MSCE) (all of these were head teachers of primary schools), one-sixth had a diploma, and almost one-quarter had a degree as their highest qualification (Exhibit 10). Exhibit 9: Number of years spent as head teacher or deputy head teacher Exhibit 10: Highest academic qualification for teachers Source: 2017 Head Teacher Questionnaire 47% 23% 30% 0% 20% 40% 60% 80% 100% 1-5 5-10 >10 % of Teachers Number of Years 60% 17% 23% 0% 20% 40% 60% 80% 100% MSCE Diploma Degree % of Teachers Highest Qualification February 2018 | ASPIRE Performance Evaluation Report 17 4.2 CHARACTERISTICS OF PRIMARY SCHOOL STUDENT RESPONDENTS Exhibit 11 shows the distribution of age among the sampled primary school learners. The mean age was 13.1 years. Learners older than 15 years comprised almost 10 percent of the sample. In an effort to gauge the socioeconomic status of primary school respondents, learners were asked to identify assets owned by their household from a choice of 10 items (e.g., radio, mobile phone, fridge, television, car, and electricity). Learners from primary school sample generally came from households with few household assets, indicating a generally poor economic background, with an overall sample median of three items per household. A substantial proportion of the students (17 percent) lived more than half an hour’s walk from their primary schools. Exhibit 11: Age distribution of students in primary grades Source: 2017 KAP Survey Questionnaire Primary school students were tested for reading in Chichewa and English. Malawi has substantial linguistic diversity but, as Exhibit 12 shows, almost all respondents spoke Chichewa at home and with their friends. Only 7 percent of respondents spoke English at home with their families, and 11 percent spoke English with their friends. Respondents also spoke several other languages in the two contexts, such as Chingoni, Chinyanja, Chitumbuka, Chiyao, and Elhomwe. 0% 5% 10% 15% 20% 25% 30% 8 9 10 11 12 13 14 15 16 17 18 20 22 % of Students Age Standard 5 Standard 6 February 2018 | ASPIRE Performance Evaluation Report 18 Exhibit 12: Main languages spoken at home and with friends (primary students) Note: Responses are not mutually exclusive. Source: 2017 KAP Survey Questionnaire 4.3 CHARACTERISTICS OF SECONDARY SCHOOL STUDENT RESPONDENTS Exhibit 13 shows the distribution of age among the sampled secondary school learners; the mean age was 16.2 years, although 9 percent of the sample comprised learners 19 years or older. A little less than two-thirds of the sampled secondary school students lived within a half an hour’s walking distance from their schools; almost 34 percent of the students lived farther away. Exhibit 13: Age distribution of students in secondary grades Source: 2017 KAP Survey Questionnaire Similar to the primary school respondents, secondary school students spoke a range of languages at home with their families and with their friends. As Exhibit 14 illustrates, almost all secondary school 95% 37% 2% 7% 6% 97% 30% 0% 11% 2% 0% 20% 40% 60% 80% 100% Chichewa Chiyao Chitumbuka English Other % of Students Primary Language Spoken at Home (n=575) Primary Language Spoken with Friends (n=577) 0% 5% 10% 15% 20% 25% 30% 35% 12 13 14 15 16 17 18 19 20 21 % of Students Age Form 1 Form 2 February 2018 | ASPIRE Performance Evaluation Report 19 respondents spoke Chichewa at home and with their friends. Almost 13 percent of the students spoke English at home, and a little more than 25 percent spoke English with their friends. Exhibit 14: Main languages spoken at home and with friends (secondary students) Note: Responses are not mutually exclusive. Source: 2017 KAP Survey Questionnaire 97% 27% 2% 13% 5% 98% 19% 1% 27% 2% 0% 20% 40% 60% 80% 100% Chichewa Chiyao Chitumbuka English Other % of Students Primary Language Spoken at Home (n=380) Primary Language Spoken with Friends (n=375) February 2018 | ASPIRE Performance Evaluation Report 20 5. FINDINGS This section presents the performance evaluation findings, organized by the six evaluation questions. Evaluation Question 1 is further organized by the three ASPIRE outputs plus the development objective to consider progress toward each activity component, as well as the overall goal. 5.1 EVALUATION QUESTION 1: ASPIRE’S PROGRESS 5.1.1 PROGRESS TOWARD DEVELOPMENT OBJECTIVE AND IMPLEMENTATION TARGETS Respondents feel there has been progress toward increasing attendance, progression, retention, and decreasing dropout rates, but monitoring data were less conclusive. Based on qualitative data, respondents across all school and community-level categories (including teachers and head teachers, students, and community groups such as school management committees, WASH committees, and PTAs) all felt there had been strong progress toward decreasing dropout rates and increasing student attendance, progression, and readmission rates after pregnancy. Respondents also felt that readmission of both boys and girls who dropped out for general reasons had also increased. This theme emerged across all three ASPIRE districts and was directly attributed to the work ASPIRE has done. Subsequent findings explore the specific ASPIRE components to which these respondents attributed progress. When asked about progress ASPIRE has made, this improvement in educational achievement was the most common theme that emerged from the qualitative data. Some respondents explicitly connected girls’ higher attendance rates during menstruation to changes in attendance patterns; for example, one community respondent in Balaka noted, “The most notable thing is … girls still attending school even in their menstruation cycles.” When speaking of these improvements in educational achievement, respondents highlighted decreased dropout rates and improved re-entry rates in particular, which by extension implies an improvement in progression rates. There has been an improvement; we had a greater number of dropouts, but with coming of ASPIRE the dropout rate has drastically dropped and some of the girls that dropped out of school back then are re-enrolling and performing well in class … which has had a tremendous impact on encouraging the Notes on Navigating Findings  Gray text boxes present baseline (2015) data.  Red, italicized text indicates an internal link. You can use these links to navigate within this report.  Italicized Quotes are illustrative of themes in qualitative data; quotes are not exhaustive. February 2018 | ASPIRE Performance Evaluation Report 21 girls to work really hard. Now they are easily and directly motivated. And the rate of early pregnancies has really decreased. —Male, Community group, Balaka We also have some girls who dropped out of school but are back in school. If we look at the graph of the students who finish their secondary school, it is increasing. —Males and females, School staff, Zomba They [girls] are back in school because ASPIRE trained mother group on how they can go and talk to the girls so that they can come back to school. —Male, Community group, Machinga The progress has been there and I thank ASPIRE for following up [with] dropout girls due to pregnancy to re-enter school following delivery. The records we get from head teacher indicate that attendance has improved. —Male, Community group, Zomba Before ASPIRE, girls’ enrollment was low. But now it is 50/50 with boys: 945 girls, 1,006 boys. —Male, School staff, Machinga While community-level respondents were highly positive about ASPIRE’s progress toward improved achievement of girls in upper primary and secondary schools, this did not emerge as a theme in qualitative data from district government officials or project-level actors. Reflecting this, monitoring data and aggregate district-level data are less conclusive in terms of ASPIRE’s progress. Exam pass rates for girls in the ASPIRE districts indicate improvement in academic achievement, particularly at the upper primary level, as shown in Exhibit 15. The two most important examinations affecting students’ progress in the upper primary and secondary levels are the Primary School Leaving Certificate Examination (PSLCE) and MSCE.1 Exhibit 15: Girls’ pass rates in national examinations (indicator 4) Exam District 2014–2015 school year (FY 2015 target) 2015–2016 school year (FY 2016 target) FY 2017 target PSLCE Balaka 55.6% (57%) 79.8% (58%) 80% Machinga 55.6% (57%) 68.3% (58%) 70% Zomba 60% (n/a) 72.15% (58%) 74.79% MSCE Balaka 52.1% (53%) 64.72% (54%) 67% Machinga 52.1% (53%) 40% (54%) 45% Zomba 40% (n/a) 40% (54%) 45% Source: ASPIRE monitoring data (2014–2015 data and FY 2015 and FY 2016 targets from FY 2015 and FY 2016 annual reports; 2015–2016 data and FY 2017 target from FY 2017 Quarter 2 Report) 1 The PSLCE is taken at the end of Standard 8 and used to make admission decisions for public secondary schools. The MSCE is taken at the end of Form 4; passage is required to earn a Certificate of Education and qualify for public higher education. February 2018 | ASPIRE Performance Evaluation Report 22 Pass rates on the PSLCE improved sharply, with year-on-year improvements of 12 to 23 percentage points among the three ASPIRE districts, exceeding the targets for that year and coming within 2 percentage points or less of the ambitious fiscal year (FY) 2017 targets. Pass rates on the MSCE were more mixed, with Balaka showing strong progress but Machinga showing equally strong slippage; Zomba remained unchanged but only began receiving ASPIRE interventions in FY 2016. All three districts were within 5 percentage points of the FY 2017 target. ASPIRE interventions would be expected to influence these exams differently: at the primary level, ASPIRE interventions supported by USAID basic education funds directly target classroom instruction that should influence exam performance. At the secondary level, implicit to the theory of change is that improving student health and reducing structural and cultural barriers to girls’ education will indirectly improve exam performance; however, funding does not directly target classroom instruction or teaching and learning materials for subjects tested in the MSCE. Dropout and repetition rates indicate mixed results, as shown in Exhibit 16. While girls’ dropout rate from 2016 to 2017 in upper primary school decreased by about 1 percentage point in Zomba, it increased in Balaka and Machinga. Repetition rates for girls and boys in upper primary increased from 2016 to 2017 in Balaka, while decreasing in Machinga and Zomba. Interestingly, while Balaka slipped in dropout and repetition rates, it was the district with the strongest PSLCE and MSCE results in 2017. These repetition rates align closely with evaluation data collected through the KAP survey, which indicate that, on average, 19 percent of learners in the population are repeating their current grade in upper primary (95 percent confidence interval: 14%–27%). KAP data indicate that there was not a significant difference in repetition rates between girls and boys, although there was an observed difference within the sample, with girls’ repetition rate in upper primary being about 4 percentage points higher. Repetition rates among secondary school students (which are not included in ASPIRE monitoring indicators) are negligible, at just 1 percent of the population, based on KAP results (95 percent confidence interval: 0%–4%). Exhibit 16: Dropout and repetition rates, Standards 4–8 District 2014–2015 school year (FY 2015 target) 2015–2016 school year (FY 2016 target) FY 2017 target Girls’ dropout rate (indicator 3) Balaka 1.9% (1.7%) 4.08% (1.4%) 1% Machinga 2.4% (1.7%) 3.64% (1.6%) 1% Zomba 4.1% (1.7%) 3.17% (1%) 1% Repetition rate of girls and boys (indicator 5) Balaka 17.80% (18%) 18.10% (16%) 15% Machinga 21.30% (18%) 19.37% (18%) 15% Zomba 23.40% (18%) 16.58% (17%) 15% Source: ASPIRE monitoring data (FY 2015 target from FY 2015 Annual Report; 2014-2015 data and FY 2016 target from FY 2016 Annual Report; FY 2017 target from FY 2017 Quarter 2 Report) Transition and progression rates, meanwhile, are harder to interpret due to strong limitations in DEMIS and ASPIRE monitoring data. Standard definitions of transition rates specify that the rate should not consider students repeating the current grade when making the calculation. However, February 2018 | ASPIRE Performance Evaluation Report 23 district data do not specify the number of students who are repeating. Concurrently, ASPIRE’s indicator that captures progression rates was altered between 2016 and 2017: the 2016 indicator refers to progression rates in Standards 4 to 8, while the 2017 indicator refers to the transition rate from Standard 8 to Form 1. This change prevents year-on-year comparison. Additionally, the data are not disaggregated by sex. With these caveats, the available data do indicate some progress. ASPIRE’s Quarter 2 report for FY 2017 notes strong improvement in girls’ and boys’ transition rates from Standard 8 to Form 1, climbing more than 8 percentage points, from a baseline of 38.5 percent in the 2014–2015 school year to just over 47 percent in the 2015–2016 school year (indicator 2). This is a large improvement in a short period, although it remains just under the FY 2017 target transition rate of 50 percent. Overall, in 2017, ASPIRE reported 76 percent of children regularly attending school, with Machinga and Zomba reporting similar rates of 74 percent and 75 percent, respectively, and Balaka reporting a higher attendance rate of 83 percent. This aligns, in general, with KAP data, which indicate that 8 percent of primary students and 10 percent of secondary students miss school “a lot” due to illness. At both levels, about 60 percent “almost never” miss school due to illness, with the remainder reporting “occasionally” missing school. To date, ASPIRE has not reported on the comparable indicator regarding the percent of children too sick to participate in daily activities, although this indicator is likely too broad to offer meaningful insight into ASPIRE’s results. Respondents attributed progress toward the development objective as the result of the work ASPIRE is doing, and indicated that the cross-sectoral design is a relevant intervention for the challenge that has yielded benefits because successes under one output are seen to contribute to successes under other outputs. Overall, ASPIRE implementation is on track to meet implementation targets. In qualitative interviews and focus groups, the most-reported reason for the perceived increase in school attendance was the resources ASPIRE has provided to students, teachers, and schools. This attribution arose most commonly among community groups and teachers, but was also noted by students and government officials. In a number of cases, respondents went on from attendance to mention improved reading skills, reflecting the interconnected way that beneficiaries naturally experience the interventions. The resources mentioned the most often included school fees for girls who were otherwise unable to attend school, followed by school uniforms and sanitary products for girls, and books and desks for schools. It is worth noting that these inputs were initiated only recently, in the 2016–2017 school year, through the DREAMS-funded block grants to all secondary schools and a subset of primary schools. Aside from resources, focus group and interview respondents across categories the most often singled out ASPIRE’s work in menstrual hygiene and sanitation as one of the most important contributions to girls’ improvements across all three outputs. The extent to which respondents drew connections across ASPIRE’s three outputs and to the development objective affirms the activity’s cross-sectoral theory of change and the activity design, which emphasizes the multiple determinants, across sectors, February 2018 | ASPIRE Performance Evaluation Report 24 that influence girls’ achievement. Responses further indicate that respondents see the three outputs as mutually reinforcing, with progress in one area supporting results in another area. Several evaluation findings below further illustrate the reinforcing and complementary nature of the different sectors ASPIRE is engaging in. For example, under Output 3, respondents noted that improvements in menstrual hygiene practices and WASH infrastructure have improved girls’ attendance at school. Similarly, findings under Evaluation Question 2, regarding school health days and the new school-based referral system for Youth-Friendly Health Services providers, show that the school is an effective entry point for health service providers to reach adolescents, thereby supporting increased access to health services. The strength of the cross-sectoral approach was best captured by a female head teacher in Balaka who explained, “The model has been successful due to enlightened multisectoral issues that holistically affect girls’ performance in schools.” Project-level actors, including USAID staff, ASPIRE staff, and subcontractors, independently affirmed the strength of the multisectoral approach in interviews and focus groups. USAID staff additionally noted ASPIRE’s broad geographic scope, which targets the entire population of schools and students in its intervention area, rather than selected schools as a strength. ASPIRE is not just working in one area. A lot of projects in Malawi work in just a small geographic space of a district. ASPIRE has the advantage that they chose to saturate, they’re working with every school in the district. I think that helps them get their message out easier. —USAID staff Exhibit 17 through Exhibit 20 shows ASPIRE’s progress to date for key implementation indicators vis-à-vis targets.2 As indicators of activities directly facilitated by ASPIRE, these indicators provide a measure of the extent to which implementation is on track. Indicators related to ASPIRE outputs and outcomes (i.e., the intended results of these activities) are covered in subsequent findings. Overall, ASPIRE’s monitoring data show that implementation is on track, meeting or exceeding the majority of its implementation targets in each fiscal year; because FY 2017 was only half-over at the time of data collection, it is not expected that ASPIRE would have fully achieved FY 2017 targets at the time of this evaluation. Exhibit 17 presents implementation progress under Output 1, and shows that ASPIRE has met, exceeded, or is on track to meet targets for literacy teacher training, with the exception of the activity’s first year. At the same time, although ASPIRE has disseminated substantial numbers of teaching and learning materials, dissemination of literacy materials (indicator 10) is lagging behind targets. For training and materials for school-based health activities under Output 2, Exhibit 18 shows that ASPIRE has made strong progress toward teacher training; despite a slow start to material dissemination, ASPIRE made progress in catching up to targets in FY 2017. Exhibit 19 shows ASPIRE’s support for school WASH and indicates that, overall, ASPIRE met or exceeded targets in FY 2016 (the first year of construction activities). At the time of this evaluation, it was too early to determine whether ASPIRE will achieve FY 2017 WASH targets. Finally, Exhibit 20 shows 2 Source for all four exhibits is ASPIRE monitoring data (FY 2015 data from FY 2015 Annual Report; FY 2016 data from FY 2016 Annual Report; FY 2017 data from FY 2017 Quarter 2 Report). February 2018 | ASPIRE Performance Evaluation Report 25 progress in training community actors in content intended to reduce structural and cultural barriers to girls’ education, and indicates that ASPIRE has met or exceeded these targets across the board. Exhibit 17: ASPIRE support for literacy instruction (Output 1) Indicator FY 2015 (target) FY 2016 (target) FY 2017 to date (target) Total Number of teachers or teaching assistants trained (indicator 6) 716 (1,260) 5,883 (2,774) 4,196 (4,476) 10,795 (8,510) Number of people trained on gender-sensitive pedagogy (indicator 7) 716 (1,630) 5,883 (2,774) 4,196 (4,476) 10,795 (8,880) Number of administrators and officials trained (indicator 14) n/a3 n/a 45 (100) 45 (100) Number of textbooks and other teaching and learning materials provided (indicator 10) 0 (189,000) 255,000 (385,920) 353,000 (443,000) 608,000 (1,017,920) Exhibit 18: ASPIRE support for school-based health activities (Output 2) Indicator FY 2015 (target) FY 2016 (target) FY 2017 to date (target) Total Number of teachers trained on code of conduct (indicator 31) n/a 1,959 (1,059) 6,002 (2,072) 7,961 (3,131) Number of sexual and reproductive health materials distributed (indicator 21) 0 (0) 3,574 (11,520) 6,062 (4,620) 9,636 (16,140) Number of joint policy advocacy events with government staff (indicator 22) 0 (2) 1 (3) 1 (3) 2 (8) Number of schools implementing school health days twice per year (indicator 34) n/a 18 (307) 157 (441) 172 (748) Exhibit 19: ASPIRE support for school WASH (Output 3) Indicator FY 2015 (target) FY 2016 (target) FY 2017 to date (target) Total Number of school sanitation facilities constructed that have:  Basic sanitation facilities (indicator 44)  Menstrual hygiene management facilities (indicator 45)  Facilities for special needs students (indicator 47) 0 28 (28) 1 (14) 29 (42) Number of liters of drinking water disinfected with point-of-use treatment products (indicator 46) 0 91,560 (12,708) 92,330 (12,708) 183,890 (25,416) Number of hand-washing stations built (indicator 49) 0 474 (715) 169 (715) 643 1,430 3 n/a refers to indicators that were not part of the original monitoring and evaluation plan; the majority of these indicators were added with the DREAMS funding in 2016. February 2018 | ASPIRE Performance Evaluation Report 26 Exhibit 20: ASPIRE support to reduce structural and cultural barriers (Output 3) Indicator FY 2015 (target) FY 2016 (target) FY 2017 to date (target) Total Number of mothers’ groups trained to create village savings and loan groups to support girl’s education (indicator 43) 70 (158) 622 (483) 617 (617) 1,309 (1,258) Number of people educated on tools, approaches, or methods for water security, integrated water resource management, or water source protection (indicator 50) n/a 5,410 (0) 0 (2,971) 5,410 (2,971) Number of PTAs supported (indicator 51) 315 (n/a)4 617 (n/a) 617 (617) 617 (617) 5.1.2 PROGRESS TOWARD OUTPUT 1: IMPROVING READING SKILLS Most upper primary students exhibit the mechanical ability to read fluently, but the higher level competency of reading with comprehension is lacking. Boys and girls exhibit similar reading fluency, but boys significantly outperform girls in reading comprehension. Weak oral English skills could partially explain the weakness in reading comprehension. Based on 2017 reading assessment data, Standards 5 and 6 students in the ASPIRE intervention districts have good English oral reading skills, both in terms of correct words per minute (the rate of reading) and the percent of words read correctly out of those attempted (reading accuracy).5 On average, upper primary students read 55 words per minute correctly (95 percent confidence interval: 50–59 words per minute), with no significant difference between boys and girls.6 This average indicates that students have developed automaticity because the rate is faster than decoding. Overall, students accurately read 86 percent of the words they attempted, with no differences by sex. This indicates that students correctly read the vast majority of 4 Indicator 51 was added in 2017. Thus, 2015 and 2016 did not have targets; ASPIRE provided the data source for those years via email on September 19, 2017. 5 For detailed explanation on the administration and scoring of these tasks, see Annex 2 - “Scoring of the Reading Assessment”. 6 In the absence of reading benchmarks, the 2015 baseline defined 30 words per minute as sufficient performance. Key Reading Terminology  Accuracy: percent of words read or questions answered correctly out of those attempted.  Automaticity: the level of reading fluency at which word recognition becomes automatic, allowing greater cognitive effort to focus on comprehension.  Fluency: the ability to read quickly, accurately, and with expression; fluency measures in this evaluation consider speed and accuracy only in assessing fluency.  Rate: the speed of reading, i.e., words read correctly per minute. February 2018 | ASPIRE Performance Evaluation Report 27 words they attempt and represents a high degree of accuracy. Combined, the rate and accuracy scores indicate that, on average, students in Standards 5 and 6 in ASPIRE intervention districts read English fluently. In addition to strong average fluency, oral reading fluency data also show a near normal distribution, as seen in Exhibit 21, indicating that the average is not being “skewed” by high- or low-performing students (“outliers”). Only about 1 percent of students tested read zero words correctly in English, and more than half read 41 or more words correctly per minute. This is notable because it means that almost all students are managing to decode or recognize words by upper primary school; there is not a large cohort of children being left behind. The second graph in Exhibit 21 disaggregates the distribution by Standard 5 and Standard 6, and shows that, as expected, students in Standard 6 read significantly faster than their peers in Standard 5 (p<0.001); the average rate in Standard 5 was 49 words correct per minute (95 percent confidence interval: 44–53 words per minute), while the average in Standard 6 was 61 words per minute (95 percent confidence interval: 55–66 words per minute). This difference of 12 words per minute represents approximately half a standard deviation. ASPIRE’s 2015 reading assessment baseline showed that, on average, upper primary students in Balaka and Machinga scored 41 percent on the English oral reading task, with boys and girls achieving the same score. However, there is insufficient documentation to determine whether this statistic refers to an accuracy score (percent correct out of those attempted) or the percent correct out of total (which would mean an average of 57 words correct per minute). In the 2017 reading assessment data, the comparable statistic for the former is 85 percent, while for the latter it is 53 words per minute; both statistics are for Balaka and Machinga only (excluding Zomba). The former would represent a strong improvement in accuracy scores over the course of 2 years, while the latter would represent an essentially unchanged words-per-minute score. However, given the lack of baseline documentation, it cannot be known which interpretation is correct. February 2018 | ASPIRE Performance Evaluation Report 28 Exhibit 21: English oral reading distribution, overall and disaggregated by grade Note: The dotted red line displays the normal distribution. Source: 2017 Reading Assessment; data available in Table 1. The 2017 Chichewa oral reading results from the reading assessment reflect a similar pattern. On average, primary students correctly read 42 words per minute (95 percent confidence interval: 39.6– 43.9 words per minute) and accurately read 93 percent of the words they attempted; there was no significant difference by sex for either statistic, reflecting parity in performance between sexes. Combined, the speed and accuracy results indicate that, on average, upper primary students in ASPIRE’s region read fluently or close to fluently in Chichewa, and have a degree of automaticity in word recognition. This task was not part of ASPIRE’s 2015 baseline reading assessment. Exhibit 22 displays the distribution of the correct words read per minute in Chichewa. Similar to English, the results are notable in their near-normal distribution and low proportion of students who read zero words correctly (about 1 percent). Notably, there was a significant correlation between zero scores in the two languages (p<0.001), indicating that most students who failed to read any words in one language also failed to read any words in the other language. The second graph in Exhibit 22 disaggregates the distribution by Standard 5 and Standard 6, and shows that students in Standard 6 0% 5% 10% 15% 20% 25% 30% % of Students Correct Words per Minute 0% 5% 10% 15% 20% 25% 30% % of Students Correct Words Per Minute Standard 5 Standard 6 February 2018 | ASPIRE Performance Evaluation Report 29 read significantly faster than their peers in Standard 5 (p=0.003); the average rate in Standard 5 was 38 words correct per minute (95 percent confidence interval: 36–40 words per minute), while the average in Standard 6 was 45 words per minute (95 percent confidence interval: 41–49 words per minute). This difference of seven words per minute represents about half a standard deviation. Oral reading fluency is an important skill because it is one of several critical components of reading with comprehension. Students who read with speed and accuracy are more likely to understand what they have read because they can focus on the meaning of the text rather than on decoding. Due to a lack of literacy benchmarks, it is difficult to establish whether students’ Chichewa and English reading fluency is grade appropriate. Bivariate analysis to determine the rate of reading associated with higher comprehension scores proved inconclusive for both languages, which could either reflect weaknesses in the reading assessment instrument or indicate that the observed reading fluency rates are not strong barriers to comprehension; the results underscore the need for literacy benchmarks and associated assessment tools that go through a rigorous piloting and validation process. Exhibit 22: Chichewa oral reading distribution, overall and disaggregated by grade Note: The dotted red line displays the normal distribution. Source: 2017 Reading Assessment; data available in Table 2. 0% 5% 10% 15% 20% 25% 30% 0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 % of Students Correct Words per Minute 0% 5% 10% 15% 20% 25% 30% 0 1-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-94 % of Students Correct Words Per Minute Standard 5 Standard 6 February 2018 | ASPIRE Performance Evaluation Report 30 Despite adequate oral reading skills, students struggled to comprehend the text they had read in both languages. Large floor effects (where students received zero scores due to not answering any questions correctly) were observed in the English reading comprehension task, as Exhibit 23 shows, with more than three-fifths of students answering zero questions correctly out of those they attempted. On average, students correctly answered 26 percent of the English comprehension questions they were asked (accuracy; 95 percent confidence interval: 20%-32%). On average, girls lagged significantly behind boys on English reading comprehension, in both correct responses out of the total possible and accuracy (p=0.009 and p=0.019, respectively). Girls’ average accuracy was 21 percent, compared with 33 percent for boys, as seen in Exhibit 24. Exhibit 23: Percentage of students who accurately completed English reading comprehension tasks Exhibit 24: Mean English reading comprehension (disaggregated by sex) Source: 2017 Reading Assessment; data available in Table 3 and Table 4. The 2017 Chichewa reading comprehension task shows similarly weak performance. On average, students correctly answered 58 percent of the questions they were asked (accuracy; 95 percent 61% 5% 12% 3% 19% 0% 20% 40% 60% 80% 100% 0% 33% 50% 67% 100% % of Students % of Questions Answered Correctly 33% 21% 0% 20% 40% 60% 80% 100% Male Female ASPIRE’s 2015 reading assessment baseline showed that, on average, upper primary students in Balaka and Machinga correctly answered 15 percent of the English reading comprehension questions, out of the total possible (boys scored 17 percent and girls scored 13 percent). Excluding Zomba, the 2017 data show that out of the total possible, students correctly answered 10 percent of the questions (95 percent confidence interval: 8%–13%); boys answered 15 percent correctly out of the total possible, while girls answered 6 percent correctly. The 2015 English reading comprehension result falls outside the 2017 confidence interval; however, because the baseline confidence interval in unknown, the significance of the difference cannot be tested. February 2018 | ASPIRE Performance Evaluation Report 31 confidence interval: 50%–66%). As Exhibit 25 illustrates, 15 percent had zero scores. Boys did significantly better than girls, with an average accuracy of 63 percent versus 54 percent for girls, as seen in Exhibit 26 (p=0.013). This task was not part of ASPIRE’s 2015 baseline reading assessment. Exhibit 25: Percentage of students who accurately completed Chichewa reading comprehension tasks Exhibit 26: Mean Chichewa reading comprehension (disaggregated by sex) Source: 2017 Reading Assessment; data available in Table 5 and Table 6. The 2017 oral English listening comprehension scores were similarly weak. On average, students correctly answered 15 percent of English listening comprehension questions. However, the average masks the strongly skewed distribution of scores (see Exhibit 27): more than half of the students (56 percent) answered no questions correctly (obtaining zero scores), and an additional 21 percent answered only one of five questions correctly, meaning that more than three-fourths of students correctly answered no more than 20 percent of the comprehension questions after listening to a story read to them in English. Statistically, boys performed significantly better on average than girls, as seen in Exhibit 28, but this difference is not practically meaningful due to the low overall scores on this task (p=0.028). This task was not part of ASPIRE’s 2015 baseline reading assessment. 15% 1% 28% 25% 4% 27% 0% 20% 40% 60% 80% 100% % of Students % of Questions Answered Correctly 63% 54% 0% 20% 40% 60% 80% 100% Male Female February 2018 | ASPIRE Performance Evaluation Report 31 Exhibit 27: Percentage of students who accurately completed English listening comprehension tasks Exhibit 28: Mean English listening comprehension (disaggregated by sex) Source: 2017 Reading Assessment; data available in Table 7 and Table 8. Finally, on the English dictation task, students had 43 percent correct responses on average, with no significant differences between girls and boys (95 percent confidence interval: 33%–53%). Only 12 percent of the students tested had zero scores on the English dictation task. Exhibit 29 presents literacy data from ASPIRE’s reporting, showing that, at the end of the 2014– 2015 school year (during ASPIRE’s first year), almost 75 percent of students demonstrated the ability to read and understand grade-level text across the three intervention districts (indicator 1). This increased dramatically in the 2015–2016 school year, when just over 93 percent of students achieved this indicator. These data are based on pass rates of the English portion of the PSLCE administered at the end of Standard 8.7 Indicator 12 is based on an internal annual literacy assessment ASPIRE administers and reported as a composite score of oral reading fluency, dictation, and comprehension tasks. Based on these data, 7 Prior to FY 2017, reporting against indicator 1 was based on a broader composite of pass rates across additional PSLCE components; consequently, the number was reported at a lower 48 percent in the FY 2016 Annual Report. 56% 21% 12% 8% 3% 1% 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% % of Students % of Questions Answered Correctly 18% 13% 0% 20% 40% 60% 80% 100% Male Female On ASPIRE’s 2015 reading assessment baseline, upper primary students in Balaka and Machinga scored 18 percent on the English dictation task, with no difference between boys and girls. Excluding Zomba, the 2017 data show that out of the total possible, students scored 40 percent on this task (95 percent confidence interval: 31%–50%). However, the baseline scoring guidelines were not documented; consequently, consistent scoring across the periods cannot be verified. Because the baseline confidence interval is unknown, the significance of the difference cannot be tested. February 2018 | ASPIRE Performance Evaluation Report 32 upper primary students showed an average literacy skills gain of 2 percent across the three districts during the 2014–2015 school year; the rate of gain increased to 4 percent in the 2015–2016 school year across all three intervention districts. Exhibit 29: ASPIRE literacy data Indicator District 2014–2015 school year (FY 2015 target) 2015–2016 school year (FY 2016 target) FY 2017 target The proportion of students who are, by the end of the primary cycle, able to demonstrate [reading] understanding, as defined by national experts (indicator 1) Balaka 73.4% (n/a)* 95.0% (93%) 94% Machinga 73.1% (n/a)* 89.6% (93%) 94% Zomba 78.1% (n/a)* 95.2% (93%) 94% Percent gain in literacy among students for Standards 4–8 (indicator 12) Balaka 2% (2%) 4.9% (2%) 6% Machinga 2% (2%) 5.2% (2%) 6% Zomba 2% (n/a) 3.1% (2%) 6% Source: Malawi National Exam Board PSLCE, as reported by ASPIRE. For indicator 1, FY 2016 targets from FY 2016 Annual Report; all other data and targets from FY 2017 Quarter 2 Report. For indicator 12, 2014–2015 data and FY 2015 and FY 2016 targets from FY 2016 Annual Report; 2015–2016 data and FY 2017 target from FY 2017 Quarter 2 Report. * FY 2015 targets are not applicable due to revision in indicator data source. In the absence of national reading benchmarks for upper primary school, it is difficult to assess this performance. With the caveats noted above, it is possible to cautiously state that 2017 results show a fair level of reading fluency, but an ongoing weakness in both reading and oral comprehension. Because comprehension is, ultimately, what allows students to extract meaning from text and “read to learn,” it is clear that substantial progress is still needed for students to be able to read with comprehension. Respondents feel that ASPIRE’s inputs are helping improve reading outcomes and educational capacity at the school level; ASPIRE’s teacher training and extracurricular activities were singled out as important contributors to success. Significant inputs from ASPIRE center on activities and resources designed to boost reading outcomes and educational capacity at the school level. These inputs range from teacher training and support with school supplies to extracurricular activities, such as reading clubs that extended school￾based learning, and facilitated reading competitions. In interviews with teachers, head teachers, and school trainers about ASPIRE’s teacher training inputs, respondents reported an increase in students’ English and Chichewa literacy skills and an overall improvement in teachers’ attitudes toward students, particularly girls. ASPIRE has trained us on teaching skills and I am one of the language teachers. The school has nine teacher[s] and six have been trained. Training covered reading fluency, vocabulary, word recognition, and comprehension. Most of our learners have improved on their reading skills, including comprehension. —Female, School staff, Balaka February 2018 | ASPIRE Performance Evaluation Report 33 Community groups, students, and government officials corroborated the contribution of teacher training to enhanced teacher capacity. The attitude of the teachers has changed positively and it is very encouraging. —Females, Community group, Zomba They also train our teachers with the skills on how they can help us gain more knowledge. —Female, Student, Zomba ASPIRE’s interventions include providing school supplies and organizing extracurricular teaching and learning activities, such as extended school-time reading clubs and reading competitions. Community groups, students, and school staff recognized these supplemental materials and activities as being important and relevant to improving students’ reading skills. There has been a great change. There has been a greater change in the numbers of learners who are selected to secondary school … I would also say that the extension of class duration has contribution to an improvement in the learners’ focus on education. —Male, Community group, Balaka Respondents across categories highlighted the importance of these teacher trainings, and teaching and learning materials for student learning outcomes. Respondents emphasized that teachers’ and learners’ attitudes have improved, and connected this to increased motivation among students and an overall more conducive learning environment. Although a statistical relationship between ASPIRE interventions and improvement in students’ reading outcomes cannot be established in this evaluation, the qualitative data show ASPIRE’s interventions to be seen as highly relevant responses to literacy challenges in upper primary school. These data provide evidence from a broad cross￾section of school and community actors that ASPIRE is contributing to strengthening literacy. 5.1.3 PROGRESS TOWARD OUTPUT 2: IMPROVING SEXUAL AND HEALTHCARE-SEEKING BEHAVIORS Basic HIV/AIDS knowledge among primary and secondary students appears to be strong, although there are still gaps in knowledge of certain modes of transmission. Findings from the 2017 KAP indicate that primary and secondary students have high levels of general knowledge about HIV/AIDS. When asked if they had ever heard of the disease, males and females reported similar levels of awareness, as Exhibit 30 illustrates. The 2015 baseline KAP data for primary students in Balaka and Machinga show that 97 percent of males and 96 percent of females had heard of HIV/AIDS. In 2017 KAP data for primary students in Balaka and Machinga, 92 percent of males and 89 percent of females had heard of HIV/AIDS. The 2015 proportion falls outside the 2017 confidence interval; however, because the baseline confidence interval is unknown, the significance of the difference cannot be tested. February 2018 | ASPIRE Performance Evaluation Report 34 When asked about more specific HIV/AIDS topics, students were the most knowledgeable about aspects corresponding directly with the Life Skills curriculum taught in schools and ASPIRE indicators. Without prompting, majorities of primary and secondary students also correctly identified the two most common methods of HIV transmission, sexual intercourse and contaminated blood, although primary students were noticeably less aware of blood as a transmission means, compared to intercourse (Exhibit 31). An important gap was evident in students’ knowledge of mother-to-child transmission, with only 9 percent of primary students and 35 percent of secondary students aware of this HIV transmission method. Exhibit 31: Percent of students who indicated whether HIV/AIDS is transmitted through: Source: 2017 KAP Survey Questionnaire; data available in Table 10 (a and b). * 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Exhibit 30). 8% 11% 65% 51% 86% 84% 31% 34% 93% 87% 98% 95% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Male Female Male Female Male Female Male Female Male Female Male Female Mother-to-Child Contaminated Blood Sexual Intercourse Mother-to-Child Contaminated Blood Sexual Intercourse Primary (n=580)* Secondary (n=381)** Exhibit 30: Percent of students who have heard of HIV and AIDS 89% 88% 99% 96% 0% 20% 40% 60% 80% 100% Male Female Male Female Primary Secondary Source: 2017 KAP Survey Questionnaire; data available in Table 9. February 2018 | ASPIRE Performance Evaluation Report 35 Indicator 18, a key measure of Output 2, tracks two key aspects of students’ HIV/AIDS knowledge:  Identifying the two major ways of preventing the sexual transmission of HIV (abstinence, and using condoms correctly and consistently)  Rejecting the two most common local misconceptions about HIV transmission (that HIV can be transmitted by mosquito bites and witchcraft). As Exhibit 32 illustrates, primary and secondary students most often identified abstaining from sex (66 percent of primary, 86 percent of secondary) and using condoms (63 percent of primary, 85 percent of secondary) as ways to prevent HIV transmission. Limiting sexual partners to one person and male circumcision were reported much less frequently. The lack of knowledge of these important prevention methods reveals important gaps in students’ knowledge of HIV/AIDS, which could be linked to gaps in the Life Skills Curriculum. At present, the curriculum focuses heavily on abstinence education; its mentions of staying faithful to one partner are far outweighed by the focus on abstinence. In addition, the curriculum does not focus on male circumcision. Consequently, ASPIRE began promoting this method only recently, with the addition of the DREAMS-supported school health days. Exhibit 32: Percent of students who stated it is possible to reduce risk of HIV transmission by: 7% 6% 9% 6% 64% 64% 65% 58% 5% 8% 36% 30% 86% 86% 89% 81% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Circumcision Limiting sexual partners to one Abstaining from sex Using a condom Circumcision Limiting sexual partners to one Abstaining from sex Using a condom Primary (n=580)* Secondary (n=381)* 00% Source: 2017 KAP Survey Questionnaire; data available in Table 11 (a and b). * 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Exhibit 30). 1 February 2018 | ASPIRE Performance Evaluation Report 36 When presented with eight common local misconceptions, more than 70 percent of primary students and 90 percent of secondary students correctly rejected them. Among those who answered that they had heard of misconceptions, the highest rejection rates again aligned with ASPIRE indicator 18, which focuses on the two most common local misconceptions—mosquito bites (73 percent by primary and 90 percent of secondary) and witchcraft (75 percent primary and 94 percent secondary) (Exhibit 33). Additionally, 2017 KAP data show that 76 percent of primary students and 96 percent of secondary students correctly identified that it is possible for a healthy-looking person to have HIV/AIDS. In ASPIRE’s 2015 baseline KAP, the highest rejection rates of common local misconceptions were different among upper primary students. These students most commonly rejected two misconceptions—holding hands with someone with HIV/AIDS (90 percent correct) and shaking hands with someone with HIV/AIDS (88 percent correct). Almost all baseline data for this question fall outside the 2017 confidence interval; however, because the baseline confidence interval is unknown, it is unclear if the confidence intervals overlap. Indicator 18 is a compound indicator that measures the extent to which students correctly answer the two knowledge questions regarding HIV transmission. At baseline, in 2015, 42 percent of upper primary students satisfied this indicator by correctly identifying two major ways of preventing sexual transmission of HIV and rejecting the two most common local misconceptions (ASPIRE 2016 Annual Report), with a target of 48 percent for FY 2017 (Quarter 2, FY 2017 Quarterly Report). However, 2017 KAP data found that among upper primary students, only 33 percent of males and 29 percent of females successfully achieved the indicator (95 percent confidence interval: 26%–40% for males, 22%–37% for females); excluding Zomba, the figure is the same for males (33 percent) and 32 percent for females. Because the baseline figure is not disaggregated by sex and the confidence interval is unknown, the significance of the difference cannot be tested. Among secondary students, in 2017, 74 percent of males and 63 percent of females met this category (95 percent confidence interval: 60%–84% for males, 58%–67% for females). ASPIRE’s 2015 baseline did not include secondary students. February 2018 | ASPIRE Performance Evaluation Report 37 Exhibit 33: Percent of students who correctly rejected common misconceptions that it is possible to transmit HIV/AIDS by: Source: 2017 KAP Survey Questionnaire; data available in Table 12 (a and b). * 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Exhibit 30) 72% 74% 77% 78% 76% 74% 0% 20% 40% 60% 80% 100% Male Female Male Female Male Female Mosquito bites Bathing in the same water as someone with HIV or AIDS Witchcraft Primary (n=580)* 93% 88% 96% 95% 98% 89% Male Female Male Female Male Female Mosquito bites Bathing in the same water as someone with HIV or AIDS Witchcraft Secondary (n=381)* February 2018 | ASPIRE Performance Evaluation Report 38 Students, teachers, and community groups reported increased access to sexual and reproductive health services for students, largely due to ASPIRE’s work in Youth-Friendly Health Services referrals and school health days, which involve HIV counseling and testing. Exhibit 34 shows the rates at which students access various sexual and reproductive health services, based on self-reporting in the 2017 KAP survey; girls and boys access these services at the same rate, with none of the differences being significant. The most common service that boys and girls at upper primary and secondary levels are accessing is HIV testing and counseling: 68 percent of secondary students had been tested (95 percent confidence interval: 52%–81%), but only 36 percent of upper primary students reported ever having an HIV test (95 percent confidence interval: 28%–44%). Students use Youth-Friendly Health Services at slightly higher rates than traditional doctors, nurses, and health clinics. Secondary students access these services more often than primary students, which aligns with this age group’s needs. The 2015 baseline KAP phrased questions regarding visiting health clinics and youth friendly health services differently, and thus data are not comparable.8 8 The 2017 KAP asked whether the student had visited a doctor, nurse, or health clinic in the current academic year. The 2015, baseline KAP asked whether they had ever seen a doctor or nurse, or visited a health clinic. In 2015 baseline KAP data, 22 percent of male upper primary students and 23 percent of female upper primary students self-reported having ever had an HIV test. In the 2017 KAP data among upper primary students, excluding Zomba, 40 percent of males and 35 percent of females reported ever having had sex (95 percent confidence interval: 26%–56% for males, 24%–47% for females). The 2015 baseline statistics falls outside the 2017 confidence interval; however, because the baseline confidence interval is unknown, the significance of the difference cannot be tested. February 2018 | ASPIRE Performance Evaluation Report 39 Exhibit 34: Percent of students who have accessed sexual and reproductive health (SRH) services via the following: 5% 5% 9% 6% 11% 4% 17% 19% 39% 34% 65% 72% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Male Female Male Female Male Female Male Female Male Female Male Female Primary* (n=580) Secondary* (n=381) Primary* (n=580) Secondary* (n=381) Primary* (n=580) Secondary* (n=381) Visited doctor, nurse, or health clinic for SRH in current academic year Used Youth Friendly Health Services in current academic year Have gone for HIV counseling and testing Source: 2017 KAP Survey Questionnaire; data available in Table 14 (a and b). * 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (see Exhibit 30). Although the lack of comparable baseline data means it is impossible to know whether this is an improvement, qualitative data from development partners and teachers indicate that respondents do perceive an increase in HIV testing, which they primarily attribute to the school health days and referral system that were added to ASPIRE with the DREAMS funding in the 2016–2017 school year. To a lesser extent, respondents connected this increase to earlier ASPIRE school and community interventions targeting students’ sexual and reproductive health practices. I have seen, with the coming in of ASPIRE, we meet a lot of adolescents in one place. Sometimes it is hard to meet adolescents because of the contraceptive part, so ASPIRE has created a place where we can reach a lot of adolescents where we can provide [HIV] services and information. —Development partner There is freedom of sexual reproductive services [family planning]; the number of girls coming for VCT [voluntary counseling and testing] has increased. Referral services are done—the head [referral agent] writes on the book such that we know that she is coming from school and are helped. Their problems are filled on the form, which can be kept confidential by them. —Male, Community group, Machinga Although qualitative data showed the majority of respondents reported an increase in access to sexual and reproductive health services for students, others noted that there is still work to be done to reach all students. This may, in part, reflect that not all schools had been reached by ASPIRE’s DREAMS components at the time of data collection for this performance evaluation. In addition, MoEST requires students under the age of 16 to obtain parental consent for HIV testing at school health February 2018 | ASPIRE Performance Evaluation Report 40 days. Parental consent is not required for students 16 and older, which is one potential reason that HIV testing rates for primary students are lower than for secondary students. Furthermore, a few NGO/development partners and community groups reported low testing numbers in Balaka, which is not a DREAMS district and, thus, does not benefit from school health days. This perception is born out in quantitative data in Exhibit 34, which show that there is still substantial room for further voluntary counseling and testing (VCT). There has been a great improvement in terms of counseling on issues related to HIV, though we do not have a good turn-up in numbers of girls going for HIV testing. —Male, Community group, Balaka Based on 2017 KAP data, students who are sexually active most commonly reported that the reason they had not gone for HIV testing was their “fear of living if HIV positive” or that “VCT was not available in their community.” These responses indicate a need to work more on desensitizing HIV in the community, and increasing HIV testing and counseling opportunities for students. In qualitative data, community groups, teachers, and students also reported an increase in access to sexual and reproductive health services through Youth-Friendly Health Services. As shown in Exhibit 34, Youth-Friendly Health Services were the second most common way students reported accessing sexual and reproductive health services in 2017 KAP data. Qualitative responses specifically pointed to the ability for students to access these services privately and securely, something ASPIRE has directly worked to strengthen by contributing to the referral system. Soon after launching the referral system, in late 2016, it was evident that the multiple referral points, including health facilities and mothers’ groups, were making student referrals difficult to track. In response, ASPIRE placed referral boxes at the schools. Since then, ASPIRE reported that 635 learners (255 boys and 380 girls) were referred through this system in FY 2017 Quarter 2, and 264 learners (110 boys and 154 girls) were referred in Quarter 3. During Quarter 3, 95 percent of referred students accessed sexual and reproductive health services, and the remainder were referred to social services. Students’ responses in the 2017 KAP survey confirmed this success. Secondary students who had accessed Youth-Friendly Health Services in the current academic year identified teachers as the most common referral source (45 percent of those who had accessed services). For primary students, teachers were the second most common referral source (26 percent), after “other.” Eighty-nine (89) percent of students reported they were “satisfied” or “very satisfied” with the Youth-Friendly Health Services received (Exhibit 35). February 2018 | ASPIRE Performance Evaluation Report 41 Exhibit 35: Satisfaction with Youth-Friendly Health Services referral Source: 2017 KAP Survey Questionnaire; data available in Table 15. Qualitative data from community members and school staff aligned with the KAP data in highlighting the success of the referral system. I also know that they have introduced a referral system for the learners to be done in privacy. —Males and females, School staff, Machinga Similar to strong knowledge of HIV transmission seen in Finding 5, 2017 KAP data show the majority of secondary students know about common contraceptive methods, with 67 percent of males and 65 percent of females having heard of contraceptives (95 percent confidence interval: 53%–79% for males, 60%–71% for females). Further, as seen in Exhibit 36, a majority of students who reported knowing of contraceptive methods could identify a variety of different types of contraceptives available without prompting, including injectable birth control (Depo), intrauterine devices (IUDs), condoms, and the birth control pill. More than 50 percent of secondary students identified each of these four types of birth control. 56% 67% 28% 27% 3% 3% 13% 3% Male Female Secondary (n=62) 58% 31% 30% 62% 6% 8% Male 6% Female Primary (n=46) Very Satisfied Satisfied Unsatisfied Very Unsatisfied 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% While overall knowledge of contraceptives was high, actual use among sexually 7 active students was more limited. Secondary students showed a higher level of knowledge of contraceptive methods than their primary counterparts. However, sexually active students—both primary and secondary—report relatively low use of condoms. February 2018 | ASPIRE Performance Evaluation Report 42 While both age groups reported knowing of injectable birth control, condoms, and the pill at high rates, secondary students reported IUD (56 percent) and sterilization (30 percent) at higher rates than primary students (19 and 7 percent, respectively). In addition, students highlighted some methods that are not seen as effective contraceptive methods, such as withdrawal and rhythm. These two responses should be flagged for further education for students. Exhibit 36: Of those who reported knowing of contraceptive methods, what types did they know about? (Five most common) Source: 2017 KAP Survey Questionnaire: data available in Table 16 (b and d). Overall in 2017 KAP data, 16 percent of primary students (24 percent male and 8 percent female) and 34 percent of secondary students (45 percent male and 23 percent female) reported ever having As would be expected, 2017 KAP data show that upper primary students were less familiar with any type of contraceptive methods than secondary students; only 30 percent of male primary students and 34 percent of female primary students reported knowing of any type of contraceptive (95 percent confidence interval: 22%–38% for males, 23%–47% for females). The 2017 rate in Balaka and Machinga districts only (excluding Zomba) is 36 percent of male primary students and 40 percent of female primary students. Although in ASPIRE’s 2015 baseline KAP, 64 percent of male primary students and 68 percent of female primary students reported knowing of any type of contraceptive methods, this comparison seems highly illogical because knowledge of HIV should have increased, if anything, due to multiple interventions by ASPIRE and others. Even if there was a decrease, it is highly unlikely to have been reduced by half from 2015 to 2017. 40% 55% 58% 52% 62% 70% 54% 71% 29% 42% 46% 59% 17% 21% 50% 62% 7% 6% 34% 26% 0% 20% 40% 60% 80% 100% Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Primary (n=184) Secondary (n=260) Primary (n=184) Secondary (n=260) Primary (n=184) Secondary (n=260) Primary (n=184) Secondary (n=260) Primary (n=184) Secondary (n=260) Condom Injectables (Depo) Pill IUD Sterilization February 2018 | ASPIRE Performance Evaluation Report 43 had sexual intercourse.9 Boys at both levels were significantly more likely to report having had sexual intercourse than girls (primary p=0.0005; secondary p=0.0023). In 2017 KAP data, students who were sexually active reported significantly higher rates of contraceptive knowledge than their peers who were not sexually active (primary p=0.0002; secondary p=0.0297). Among these sexually active students, 78 percent of secondary students had heard of any type of contraceptive (95 percent confidence interval: 66%–86%), but only 55 percent of primary students had heard of any type of contraceptive (95 percent confidence interval: 46%–64%). This low rate of knowledge among sexually active primary students is concerning because it indicates that half of primary students are at high risk of pregnancy. While knowledge of contraceptives was high, actual use among sexually active students was limited. Among primary and secondary school students who reported being sexually active, condoms were essentially the only contraceptive method used (one person reported using the pill and one the rhythm method). Exhibit 37 presents 2017 KAP data, indicating that only 28 percent of upper primary students reported using a condom at last intercourse (95 percent confidence interval: 18%– 40%); the figure was 56 percent among secondary students (95 percent confidence interval: 45%– 67%). Differences between sexes were not significant at either level. Although current condom use is low, a greater proportion of sexually active students indicated they planned to use condoms in the future. Among all students (not just those who are sexually active), the proportion planning to use condoms in the future is even higher than among the subpopulation of just those students who are sexually active (data presented in Annex 3, Table 19 (a and b)). Male students reported the intent to use condoms in the future at a significantly higher rate than females (primary p=0.0020, secondary p=0.0406). Students were not asked about future plans to use other types of contraceptives. 9 95 percent confidence intervals: males, upper primary: 16%–33%; females, upper primary: 5%–15%; males, secondary: 30%–61%; females, secondary: 13%–37%. At baseline, in 2015, 25 percent of male students and 12 percent of female students in Balaka and Machinga districts reported ever having had sex. For Balaka and Machinga districts only (excluding Zomba), the 2017 KAP data indicate that 30 percent of upper primary male students and 9 percent of upper primary female students reported having ever had sex. Baseline results lie within the 2017 confidence intervals, although the baseline sample for this question (n=104) is considerably smaller than the 2017 sample (n=580); however, because the baseline confidence interval is unknown, the significance of the difference cannot be tested. According to the 2015 baseline KAP, among upper primary students, 30 percent of males and 20 percent of females reported having used a condom the last time they had sex. In 2017 KAP data for Balaka and Machinga only (excluding Zomba), 32 percent of both males and females reported having used a condom the last time they had sex; however, differences in question wording limit comparison.8 February 2018 | ASPIRE Performance Evaluation Report 44 Exhibit 37: Condom use among sexually active students Source: 2017 KAP Survey Questionnaire; data available in Table 17 and Table 18. ASPIRE’s use of activity cards, which educate students on many of the topics reported, may have had an impact on the high level of knowledge on different types of contraceptives, because one of the cards in the package explicitly focuses on contraceptives. ASPIRE’s work with Youth-Friendly Health Services might also be contributing to knowledge of contraceptives because this is, typically, a key part of Youth-Friendly Health Services interventions. However, gaps in knowledge regarding contraceptives reflect another challenge, as ASPIRE staff mentioned. Primary and secondary school teachers cannot teach about contraception and it is not an explicit focus of the existing Life Skills Curriculum, hindering ASPIRE’s ability to teach about this issue. ASPIRE does not have the mandate to change the curriculum, but only to buttress the existing curriculum. 5.1.4 PROGRESS TOWARD OUTPUT 3: REDUCING STRUCTURAL AND CULTURAL BARRIERS ASPIRE has made substantial progress in reducing structural barriers to girls’ education, particularly in regard to hygiene and sanitation. As one method for reducing structural barriers to education and increasing girls’ school attendance, ASPIRE has focused on improving girls’ hygiene and sanitation at school. Across all informant groups, the most recognized activity has been building menstrual hygiene facilities. As a result of its work on improving WASH infrastructure (Exhibit 19 above), ASPIRE has already met 60 percent of its 2017 goal of reaching 36,000 people in FY 2017, having provided 22,107 people with improved sanitation as of Quarter 2. Nonetheless, school checklist data collected during 27% 28% 59% 52% 40% 58% 70% 45% 0% 20% 40% 60% 80% 100% Male Female Male Female Male Female Male Female Primary (n=97) Secondary (n=117) Primary (n=97) Secondary (n=117) % of students who used a condom at last intercourse % of students who plan to use a condom in the future February 2018 | ASPIRE Performance Evaluation Report 45 this evaluation indicate that only 37 percent of the sampled schools had menstrual hygiene management facilities. This underscores that the need for WASH infrastructure investment remains high and is beyond what can be accomplished with the funding provided to ASPIRE; Finding 25 discusses this in additional detail. Exhibit 38: ASPIRE’s support for hygiene and sanitation Performance indicator District Achievement to date (Target) Number of people gaining access to a basic sanitation as a result of U.S. Government assistance (indicator 48) Balaka 10,125 (12,000) Machinga 11,982 (12,000) Zomba 0 (12,000) Total 22,107 (36,000) Source: FY 2017 Quarter 2 Report. Based on school checklist data, ASPIRE-supported facilities were observed to be of high quality, with a drop hole for proper disposal of sanitary pads. All but one facility included a private hand- and pad-washing facility separate from the hand-washing station outside the latrine. All but one school visited had latrines available specifically for girls; the school that did not have a dedicated girls’ latrine had no latrines for any students. ASPIRE reported encouraging those schools it could not support to pursue projects on their own that could bring them up to standards. In 2017 qualitative interviews and group discussions, when data collectors asked about ASPIRE’s contribution to reducing structural barriers, respondents of both sexes most often referred to the presence of these new hygiene facilities. School staff, community groups, and students reported that girls felt more comfortable and willing to come to school during menstruation due to these facilities. KAP data confirm this result. In 2017, 88 percent of upper primary girls and 95 percent of secondary female students reported coming to school when they were menstruating. Further bolstering this finding, 2017 KAP data show that 86 percent of primary students and 91 percent of secondary students felt comfortable using the toilets at school, with no significant difference by sex. In qualitative data, respondents also reported that, due to its success, this model is beginning to be replicated by other schools. Exhibit 39: Percentage of girls who come to school when menstruating Exhibit 40: Percent of students who feel comfortable using school toilets % 86% 91% 92% ale Female Male Female rimary (n=580) Secondary (n=381) 86 0% 20% 40% 60% 80% 100% M P Source: 2017 KAP Survey Questionnaire; data available in Table 19 and Table 20. 88% 95% 0% 20% 40% 60% 80% 100% Primary (n=112) Secondary (n=179) February 2018 | ASPIRE Performance Evaluation Report 46 The issue of menstrual hygiene, the toilets built by ASPIRE is something worth noting. Last week, I had two head teachers from Madzimaela and their PEA [Primary Education Advisor] to see how the room was constructed and they want to construct using their own resources, which means that the locals have welcomed it. —Male, School staff, Machinga Yes, there is a change. Because of the changing rooms we have in this school and girls are no longer staying at home even if they are doing their period. —District government official, Zomba ASPIRE has really helped the girls not to stay at home even if they are doing their monthly period because of the changing rooms we have at this school (R5). And we have buckets where we put water that we use to wash hands [after using the] toilets (R3). —Males, Students, Zomba The most important thing is the changing rooms we have here at school. Girls are now coming to school even when in menstruation. Previously, when girls start having their monthly period, they stopped coming to school for fear of soiling their clothes and then boys would laugh at them. With change rooms, girls do not stop coming to school because of menses. —Female, School staff, Zomba As part of this work, ASPIRE incorporated improvements in water and hand-washing facilities, providing six hand-washing buckets with taps to all schools in the intervention districts. This additional focus on hand-washing stations helped girls feel safe and comfortable using the facilities. Teachers, community members, and NGO/development partners reported that improved hand￾washing facilities encourage girls to use the toilets and increase their attendance rates. The 2017 KAP data showed 88 percent of primary students and 94 percent of secondary students reported having clean water for hand washing. However, 80 percent of primary and 72 percent of secondary students reported that the water pumps were sometimes broken. Here is progress in girls’ health, mainly in the facilities that help improve the issue. For example, now the latrines have water and other hand-washing facilities just outside them, allowing whoever uses the toilet to wash their hands clean after helping themselves. There is a change in the students’ behavior, such as reduction of cases of dropouts and early marriages. —Male, Community group, Machinga I know that ASPIRE is promoting WASH activities and these are working very well. ASPIRE provided buckets, which are placed at strategic places where learners use to wash their hands. — NGO/Development partner ASPIRE also worked with community groups, including WASH committees, mothers’ groups, and teachers to increase education on the importance of hygiene management. As of FY 2017 Quarter 2, ASPIRE reported educating more than 25,000 people on tools, approaches, and methods for water At baseline, in 2015, ASPIRE’s KAP data show that 59 percent of female primary students reported coming to school when they were menstruating. Excluding Zomba, in 2017 KAP, 86 percent of upper primary girls reported coming to school when they were menstruating (95 percent confidence interval: 77%–92%). The 2015 data falls outside the 2017 confidence interval; however, because the baseline confidence interval is unknown, the significance of the difference cannot be tested. February 2018 | ASPIRE Performance Evaluation Report 47 security, integrated water management, and protection of water sources. In addition, students and teachers reported the value of ASPIRE’s work in teaching them more about the importance of hygiene. ASPIRE has helped us know issues of health, hygiene, and reading, even writing (R2). —Female, Students, Balaka I have been involved in raising awareness on hygiene after using the toilet for both learners and teachers. I have also been involved in construction of change rooms for girls through the school improvement fund. —Female, School staff, Balaka Finally, school staff, community groups, and students reported access to sanitary pads in schools as an important aspect of ASPIRE’s hygiene and sanitation work. Teachers, students, and community groups reported that the availability of sanitary pads has been a key contributor to keeping girls in school. For information on the work of mothers’ groups in sanitation, see Finding 19. The sanitary pads in the shops. We buy and keep them here in school; we give to all the girls in school when having their monthly periods. I tell them not to hesitate to inform me so that they do not go back home and miss classes. —Female, School staff, Balaka Teachers have assured us to approach them whenever we have problems related to menstruation (R3, R4, R5). —Female, Students, Balaka According to respondents, ASPIRE has made good progress in reducing cultural barriers to girls’ education and health through policy advocacy at the national level, and at the local level, capacity building and operationalizing strategies that help safeguard girls. ASPIRE has been successful in improving secondary school attendance rates by training and sensitizing community members and school staff on the importance and relevance of many health and education topics, such as menstrual hygiene and other girls’ health considerations, reinforcing the readmission policy, and advocating for stricter bylaws for teachers and community members assaulting or impregnating girls. Respondents noted that a key accomplishment has been ASPIRE’s work with mothers’ groups to reinforce the readmission policy of 1993, which reversed the practice of expelling girls who become pregnant. Mothers’ groups are now perceived to be an important community force for promoting attendance and readmission. Mother groups are playing a very good role in encouraging a girl child to go back to school. The learners were trained on how they can abstain and avoid contracting sexual[ly] transmitted diseases. We also have some girls who dropped out of school, but are back in school. If we look at the graph of the students who finish their secondary school, it is increasing. —Males and females, School staff, Zomba ASPIRE has done well with readmission policy through establishment of mothers’ groups who follow up boys and girls who dropped out of school. Through menstrual hygiene management, ASPIRE has February 2018 | ASPIRE Performance Evaluation Report 48 reduced absenteeism due to menstruation, which is important, but I would not know the exact policy. WASH policies I am sure are there, too, and ASPIRE has encouraged cleanliness. —Female, School staff, Balaka Where ASPIRE has done very well is to get girls and some boys back to school through mother[s’] groups. This has supported or reinforced the Re-Admission Policy. —District government official, Machinga The historic challenge of girls being sexually assaulted by teachers and community members was viewed to contribute to higher dropout rates. Respondents reported the need for harsher punishments and stronger policy responses to these offenses. Stakeholders recognized ASPIRE’s role in promoting harsher penalties. I can suggest that ASPIRE should support the enactment of laws to give tough policies for males that will be responsible for teenage pregnancies, be it teachers/fellow learners, as mostly there are alternative private sector education system[s] that harbor such conduct. —District government official, Balaka ASPIRE has worked with communities and necessitated introduction of tougher penalties for those involved in sexual violence/abuse, early marriages. —Female, School staff, Balaka ASPIRE has helped in training chiefs on banning early marriages and the benefits of re-admitting teen mothers to school. —Male, Community group, Balaka In a tangible sign of progress resulting from this advocacy work, ASPIRE reported that all schools in its three districts have now operationalized strategies to help safeguard girls, as shown in Exhibit 41. Exhibit 41: Schools operationalizing strategies to safeguard girls Indicator FY 2015 (target) FY 2016 (target) FY 2017 to date (target) Total Number of schools operationalizing school-based strategies, guidelines or policies to safeguard girls (indicator 40) 315 (175) 622 (374) 617 (617) 1,554 (1,166) Source: ASPIRE monitoring data (FY 2015 data from FY 2015 Annual Report; FY 2016 data from FY 2016 Annual Report; FY 2017 data from FY 2017 Quarter 2 Report). Teachers and school trainers reported progress in breaking down harmful cultural barriers linked to girls’ health and education in many areas. These include religious and village leaders being more aware of the importance of family planning and safe circumcision, planning chinamwali (initiation ceremonies10) around the school calendar, students seeking healthcare through school referrals and Youth-Friendly Health Services, and strengthening bylaws or creating new ones that ban early marriages and punish parents and sex offenders. 10 Chinamwali is a traditional initiation ceremony both boys and girls participate in as a rite of passage in Malawi. February 2018 | ASPIRE Performance Evaluation Report 49 Chiefs and mothers’ groups have also been trained. Chiefs, for example, they were sensitized on bad cultural practices because chiefs have the power to stop or not. Since they were trained, the chiefs have initiated development of the bylaws. —Female, School staff, Balaka ASPIRE formed linkages with community leaders and has so far been successful, with development of bylaws a at community level to deal with offenders of teen pregnancies and parents that give their children to early marriages. —Female, School staff, Balaka ASPIRE has done a lot in this regard to going further from sensitizing the community, to even educating the custodians of practices like initiation ceremonies (angaliba, anankungwi) to adjust their calendar in symphony to the school calendar such that these practices, though culturally significant, should not hinder the girls’ education, and even adjusting themes and songs of their practices that encourage sexual indulgencies (R3, R1). They were also sensitized to change practices of using single razor blade for all initiates, in the face of HIV (R5). —Males and females, School staff, Balaka Noting the central role initiation ceremonies play in child and adolescent health practices, stakeholders gave particular attention to this area. To eliminate harmful cultural practices, stakeholders encouraged ASPIRE to work with chiefs to discourage initiation ceremonies and early marriages, and penalize parents who encourage either. Some respondents even reported—and viewed positively—that local chiefs have started fining families whose daughters become pregnant, while others fine families who do not send their children to school. A, some authorities should encourage our chiefs to establish bylaws on parents who allow their children to get marriage, or who organize marriage for their children with older men. B, need to have stiff punishments for parents who infringe the rights of their children. C, abolish bad cultural practices such as chinamwali because these encourage early sex among boys and girls because from chinamwali they are encouraged to go and sleep with a boy or a girl … Fisi (hyna) culture is still being practiced chinamwali for boys is, however, improving. —Male, School staff, Machinga In FY 2017 Quarter 2, ASPIRE conducted school-based review meetings to assess the extent to which local leaders were dealing with harmful practices. Successes highlighted during this exercise include enactment and enforcement of bylaws against early marriage, school dropouts, and harmful traditions. In Quarter 3, ASPIRE reported similar success stories of communities creating or strengthening the existing bylaws. ASPIRE also supported school community groups this quarter to implement their action plans, which had been previously developed with ASPIRE support. Innovative, localized interventions reported [in] quarter three include: community enacted by-laws to eliminate harmful cultural practices. An interesting case is the story of GVH [Group Village Headman] Mposela in Mpilisi zone, where by￾laws have significantly contributed to reduction of teenage pregnancy and early marriages. —ASPIRE FY 2017 Quarter 3 Report, p. 12 These successes, identified in qualitative data, align with patterns in the 2017 KAP data, which indicate similar responses from boys and girls regarding many of the more common cultural barriers that can impede girls’ education. There was no significant difference between the percentage of boys February 2018 | ASPIRE Performance Evaluation Report 50 and girls in either upper primary or secondary level who reported being read to at home (43 percent primary, 17 percent secondary), reading on their own at home (87 percent for both primary and secondary), taking books home from school (72 percent primary, 60 percent secondary), being teased at school (23 percent primary, 15 percent secondary), or being punished at school (62 percent primary, 55 percent secondary). Among those reporting being punished at school, there was no significant difference between sexes in the percent reporting experiencing corporal punishment (6 percent primary, 3 percent secondary). Additionally, there was no significant difference in how boys and girls perceived their English and Chichewa teachers’ attitudes toward class: at both levels, 87 percent of students described their Chichewa teacher as having a “nice” or “happy” attitude; for English, 86 percent of primary and 87 percent of secondary students described their teacher in those terms. Finally, among upper primary students, there was no significant difference between boys and girls in the percentage of those who reported having access to reading materials at home (70 percent) or receiving help with their homework at home (58 percent). 2017 KAP data did reveal some differences between boys and girls. Most importantly, there was a significant difference between the percentage of those who reported feeling safe walking to school (primary: p<0.001, secondary: p=0.001). Eight-one (81) percent of upper primary boys felt safe walking to school, compared to just 68 percent of girls. In secondary school, 80 percent of boys felt safe walking to school, compared to 65 percent of girls. Additionally, at secondary school, there was also a significant difference between boys and girls in the percent of those who had access to reading materials at home (72 percent for boys, 61 percent for girls, p=0.019), and those who reported receiving help with homework at home (37 percent for boys, 51 percent for girls, p=0.006). Students reported that next to teachers, families are an important positive influence on their health and education practices, although many students also noted negative influences from family. Respondents spoke in general terms and mentioned ASPIRE’s work with families only infrequently. When asked about different sociocultural forces that affect girls’ education, students and teachers emphasized the importance of parents’ and families’ involvement in health issues. Students affirmed the importance of family involvement, reporting that family members are the first people students turn to for education, guidance, and support about sexual and reproductive health issues. My parents told me that I should abstain from sexual intercourse and that if you are not feeling well, should go to hospital for diagnosis (R5). They encouraged us to be involved in health groups so that we should receive these messages (R3). My grandparent has been advising me that when [I] am away from home, [I] should avoid sexual intercourse because it brings problems in your life (R4). My parents tell me if I fail to abstain, I should use a condom (R1). My parents at home tell me to cook and I help my siblings with household chores (R2, R3, R5) —Males, Students, Machinga We have been told by our guardians and relatives that we should not start using some contraceptives until we become older, because it makes a woman barren, and that sex between partners that have vast age differences and maturity damages the younger ones’ fertility (R5, R6). They have also advised us to avoid being close to boys and in odd hours (R3), whereas some of us have not been told anything February 2018 | ASPIRE Performance Evaluation Report 51 from our home, but have heard from peers (R5). In terms of menstrual hygiene, our families encourage us to bath[e] multiple times a day when they start their menstrual cycles (R3, R4). They have also told us to continue blood tests periodically. —Females, Students, Balaka Not all beneficiaries reported that they were receiving positive counseling from family members, and some reported they were not being counseled enough, pointing to the strength of the existing cultural barriers that hinder ASPIRE’s efforts in achieving its results. They should give us advice very frequently so that we can be more aware of these issues (R3). They should also be more open to us and tell us clearly about these practices (R2). —Females, Students, Zomba I think our parents should continue to counsel us more so that we can be good boys (R1, R7). But most of the parents think that we are children and fail to tell us in details things concerning sexual activities so maybe if they can ask our neighbors or their friends to be counseling us … (R7). Our parents should be fully engaged in sensitization meetings that happen within the community so that they can be up to date with information and then mentor us (R5). —Males, Students, Machinga Beneficiaries also discussed their parents’ central role in their broader education and learning habits. The majority of students reported that their parents were supportive either by encouraging them to learn (helping with homework and practicing reading) or being cognizant of the amount of housework they assigned their children. Our parents and guardians have helped us learn though it [is] periodic because they have their own things to do, but if there is homework they help us where we are stuck (R2, R3, R1). —Females, Students, Balaka In addition, my father taught me to read (R1) and also the relatives frequently checked my past papers from part-time to see my progress. My brother selected words which I do not know and asked me to read. He congratulated me if I passed and if I failed, punished me with a light punishment like sweeping. After sweeping, [I] was trying again to read until I pass[ed] and then [he] released me. The father was checking my exercise books, where I have failed, [he] could teach me how should have been done (R4). My relatives encouraged me to go to school and provided necessities like soap to wash my clothes so that I should be going to school (R2, R3, R1). They also promised that I will go to an expensive school if I pass (R2). —Males, Students, Machinga A smaller group of students reported that their parents did not have enough time to help with their studies, assigned time-consuming household chores, or were not supportive at all. Yes, there are a lot of things that make our parents not to support us with reading sometimes. Sometimes, when we reach home from school, we find that our parents are not yet back from their businesses and their work and they come late hours saying they are very tired as a result they don’t support us with our studies (R1). Some of our parents, they did not go to school, as a result it’s very difficult for them to help us because they don’t have any knowledge on how they can help us (R3). — Females, Students, Zomba February 2018 | ASPIRE Performance Evaluation Report 52 Sometimes parents say it’s a waste of resources for us to go to secondary schools because there are no longer job opportunities (R6, R3). —Males, Students, Machinga 5.2 EVALUATION QUESTION 2: INTEGRATION WITH OTHER USAID-FUNDED ACTIVITIES ASPIRE is integrating well with numerous other USAID-funded activities, particularly around school health days, and sexual and reproductive health service provision, including HIV testing and counseling. ASPIRE’s annual and quarterly reports have identified several integrated activities in health and education over the past 3 years. Exhibit 42 summarizes those activities and partners. Exhibit 42: ASPIRE’s integration with USAID partners When asked about ASPIRE’s success in integrating with other USAID-funded activities, community- and project-level respondents’ responses varied. School staff, mothers’ groups, and community groups most commonly cited ASPIRE’s consortium as an example of successful integration of activities. FAWEMA was the consortium partner most commonly referenced as working to eliminate stigma and discrimination, followed by MIE’s work on training teachers and CRECCOM’s work in community mobilization. The Malawi Institute of Education, CRECCOM, and FAWEMA have created a very good platform to all work with ASPIRE. The mutual training of staff has also proved to be very important. —Male, Community group, Balaka Reading and Education Activities  Early Grade Reading Activity (RTI): Jointly planned school and zonal reading fairs.  Strengthening Early Grade Reading in Malawi (SEGREM) (Malawi Institute of Education): Working collaboratively to pilot teachers’ English language proficiency training in 10 clusters in Balaka, Machinga, and Zomba districts. Health Activities  Support for International Family Planning and Health Organizations 2 (SIFPO2) (BLM/Marie Stope in Zomba and PSI in Machinga): Providing contraceptives, and HIV testing and counseling on school health days.  Dignitas International: Providing HIV-positive motivational speakers for school health days.  One Community (Johns Hopkins University/Center for Communication Programs): Supporting mothers’ groups in collaboration with ASPIRE and working with ASPIRE on case management of students found HIV-positive. Where learners are found to be HIV positive, they are referred to One Community for household assessments. February 2018 | ASPIRE Performance Evaluation Report 53 Outside of the consortium, few community-level respondents could identify NGOs that were or were not funded by USAID; however, community members referred to ASPIRE’s overall willingness to coordinate with other NGOs. ASPIRE appears to have a good working relationship with other partners; for instance, there are complementary activities being done by other NGOs (e.g., Camfed), on girls’ education empowerment. So far, no challenges have appeared in between ASPIRE and such partners. —Male, School staff, Balaka There is a great future for these partners, because their programs do not collide and organization has its own structure in every school. —Female, School staff, Machinga Project-level partners (ASPIRE staff and NGO/development partners) reported that the most successful integration of activities has occurred during school health days. For these events, ASPIRE works with SIFPO2, implemented by BLM/Marie Stopes and PSI, which provides HIV testing and counseling services, and sexual and reproductive health resources and contraceptives, as well as Dignitas International, which provides HIV-positive guest speakers to give motivational speeches and talk with students. ASPIRE provides health talks on sexual and reproductive health and HIV issues more broadly and coordinates the events, which grant these USAID partners much closer access to schools. ASPIRE also partners with One Community to provide case management for students who have identified gender-based violence cases during VCT services and have tested positive for HIV. NGO/development partners reported that their partnership with ASPIRE has expanded their access to schoolchildren, making them more effective in reaching their targets. Traditionally, health NGOs have had difficulty reaching students due to limitations on testing on school grounds and the need for parental permission prior to testing. ASPIRE and its partners worked collaboratively to ensure that testing was offered near, but not on, school grounds. I saw they were able to identify cases of gender-based violence in schools which are referred to other partners and management. I have seen, with the coming in of ASPIRE, we meet a lot of adolescents in one place. Sometimes it is hard to meet adolescents because of the contraceptive part so ASPIRE has created a place where we can reach a lot of adolescents where we can provide service and information. —NGO/Development partner Although community members did not specifically mention integration across USAID-funded activities when discussing school health days (as seen in Finding 6 under Output 2), the model of these school health days was one of the main successes they reported for increasing girls’ positive healthcare-seeking behavior; substantial integration across USAID activities, which is a feature of these events, is fundamental to the model. They’ve had 3 quarters now under DREAMS Initiative where they’ve worked with the service delivery partners to really bring services as close as possible to the schools in what we call school health days. And just in 3 quarters alone, over 20,000 students were tested and then over 500 teachers. Now you can definitely see that force creating demand and now they’re creating a safe space. I personally think February 2018 | ASPIRE Performance Evaluation Report 54 it is the best, students really do queue up to receive services. And also, services delivery partners to bring students who are living positively with HIV and can encourage students to get tests; so, I think that’s been really encouraging. — NGO/Development partner A USAID staff member discussed the broader relationships that ASPIRE has established with USAID-supported service delivery partners in schools, even suggesting that ASPIRE be seen as an example of integration for other development partners: I think I mentioned to you there are other school partners who are in the same schools and ASPIRE has really been working with them, and we’ve encouraged them [to] share curricula, so they don’t have to recreate it. We’re very happy we can recommend to other partners to build off of what ASPIRE has started, and not build from scratch. So, I’d say they’re working very well with other kinds of partners in the districts. —USAID staff Shared objectives and complementary activities with other USAID-funded projects helped enhance integration. USAID’s joint planning meetings and assistance in coordination also facilitated greater integration. Although community-level respondents had difficulty identifying which partners were funded by USAID, information from ASPIRE reports (see Finding 10) makes it clear that most of these respondents were referring to ASPIRE’s work with other USAID-funded projects. (Moreover, the inhibiting and facilitating factors that these respondents reported align with factors traditionally found in development programs.) At the project level, respondents were better able to identify and comment on inhibiting and facilitating factors related to integration with specific USAID partners. Most community respondents discussed ASPIRE’s integration more generally and described common objectives among organizations as the key facilitating factor in this integration. These respondents reported that ASPIRE takes advantage of other programs that have shared objectives and complementary activities. One respondent noted that ASPIRE is integrating with NGOs that are doing similar activities (e.g., camps, formulation of bylaws, and sanitation and hygiene), while another highlighted ASPIRE’s work with NGOs doing complementary activities (e.g., providing water for latrines while ASPIRE constructs toilets for girls, or boosting literacy while ASPIRE works on reading skills). So, one NGO does literacy boosting while ASPIRE does reading skills, which are the same thing … ASPIRE gave us momentum such that during the awareness campaigns for ASPIRE, we would discuss issues about child protection which is being advocated by [the other NGO]. —Male, School staff, Machinga There is proof of coordination between partners. Though there are complementary actions by other partners, all work in tandem. —Female, School staff, Balaka Project-level staff, NGO/development partners, and government officials all emphasized communication as a key to ensuring good collaboration. Although still a challenge (see Finding 13), USAID and NGO/development partners reported activities aimed at increasing communication February 2018 | ASPIRE Performance Evaluation Report 55 through various levels—at joint annual and quarterly planning meetings with USAID, through the district government structures and the collaboration that occurs there, and through field-level communication for implementation of activities. The most successful method reported was the USAID joint planning meetings, where partners can share best practices and achievements, and plan together to avoid duplication of efforts. The most principal factor is joint planning and where possible execution of ASPIRE activities by the three partner organizations implementing ASPIRE. For example, all partners conduct joint annual and quarterly planning meetings. During these meetings, that is where experiences and lessons learned are shared, and find common ways to improve and support each other (way forward mapped). Again, a respectable number of ASPIRE projects officers from the three partners share same office space/complex and usually go into the field together when need be. — NGO/Development partner Mission—through our Program Office, they organize this integration meeting. At the same time, the district is organized so they managed to do this through the district government structures. That is what has worked more and achievement of that. They meet at the district and that collaboration at district level has allowed it to achieve some of the objectives. —USAID staff Development partners interviewed also mentioned that shared goals and responsibilities were important for facilitating integration among partners. One development partner reported that a newly formed memorandum of understanding has facilitated integration with ASPIRE, as has the introduction of a PEPFAR DREAMS coordinator in Lilongwe. The coming in of the coordinator to coordinate all of the DREAMS partners has brought sanity to the coordination. Also, the MOU [memorandum of understanding] thing should be taken deeply into the system even though most of them were not used to the system, but it would be the best way to coordinate activities throughout the DREAMS. This also enhanced coordination, but also coming in of coordinator, it helped, but also supporting the issue of MOU, which I really appreciate. — NGO/Development partner While recognizing progress, implementers identified communication as the primary challenge to integration of USAID partners’ activities; to a lesser extent, implementers pointed to competition for beneficiaries as a challenge. Inadequate communication was the most common challenge to integrating USAID partners’ activities, highlighted across respondent categories. This challenge particularly emerged with reference to the consortium of DREAMS partners, but was also mentioned with reference to the team of ASPIRE implementing partners (CRECCOM, FAWEMA, MIE, and Save the Children). At the broad level of integration across USAID activities, NGO/development partners reported poor communication between partners as the main factor inhibiting integration, noting that programs sometimes operate in parallel because they are not communicating. However, partners pointed to this challenge in communication as not only an ASPIRE problem, but also a broader problem with February 2018 | ASPIRE Performance Evaluation Report 56 all NGO partners not using District Education Committee (DEC) meetings as much as they should, as well as difficulty with DREAMS partner communication at start-up. The most important opportunity missed by all partners (at district level) is that instead of using the [DEC meetings] for planning, that is not done. Each partner seems to compete for no good reason at all. As you are aware, all partners and projects are members of [DEC meetings] at district level. Only if these DEC meetings are properly used—for example, as joint planning meeting, annual and quarterly plans—partners would be able to identify sister partners/projects and collaborate in advance on how to work together and achieve the intended goals. — NGO/Development partner Partners within DREAMS consortium—lack of dialogue and communication with how we can work together as a consortium, rather than focusing on our own indicators within the project. — NGO/Development partner Some Government of Malawi officials echoed this sentiment, noting that partners seem to conduct similar activities, but work vertically. It was also unclear to some of these respondents how ASPIRE collaborates with the different partners. The NGO Education Coordinating Forum is useful in this regard because it helps government officials understand the activities of different partners and create action plans. You know most of these partners have a common goal, they target the same learner, implement similar activities but work vertically. —District government official In terms of competition, respondents reported that each organization focuses on its own indicators, particularly DREAMS partners. NGO/development partner staff noted that, although the national USAID structure was working to bring partners together, field-level competition to achieve targets results in multiple organizations pursuing the same beneficiaries during implementation. NGO/development partners suggested a need for better planning to avoid this problem, not only among partners, but also at a higher level, by USAID; one partner specifically observed that the drive for greater collaboration had to come from above. Tight work schedules for the projects mostly hinder the coordination with other partners, because every partner has to been seen producing results according to the workplan, without considering the other partners for USAID in any case. — NGO/Development partner Improve on coordination of activities and planning. You find that we “fight” for the same target group and in the process, frustrate one part of the project. — NGO/Development partner NGOs work as competitors and not partners. Transparency is a challenge among partners. We submit our budget at ADC [Area Development Committee] meeting, but some NGOs don’t submit their budgets. Weak ADC structures—reporting mechanisms from stakeholder is not there. We need someone from a big office at this level to push us; this would help us to work as partners and not competitors. —NGO/Development partner To a lesser extent, communication challenges were noted as another barrier to activity integration within the ASPIRE implementing team (CRECCOM, FAWEA, and MIE). One partner reported February 2018 | ASPIRE Performance Evaluation Report 57 challenges in everyday communication, noting that ASPIRE field staff were slow to respond to emails and citing difficulty with overall internal communication that might have caused delays in implementation. Another ASPIRE partner noted that competing work schedules and funding delays had previously undermined a harmonized implementation plan. Partners also complained that some lack of transparency on ASPIRE’s part made it difficult to understand where they are and what they are doing, and avoid duplication of efforts. Sometimes it is confusing, because it is ASPIRE but has [a] DREAMS component within ASPIRE. So, sometimes it is difficult to understand what they are doing within the community, within the schools, so we can work together. —NGO/Development partner One of the challenges could be competing work schedules sometimes due to delayed funding. At times, partners’ activities “crash.” The issue is that when funding delays, partners carry forward some of the activities. This means that the harmonized implementation plan becomes absolute and activities are carried out just to fulfill the schedule. — NGO/Development partner At the community level, this lack of communication was the most reported by teachers and head teachers. Teachers reported that the lack of communication led to either a duplication of efforts by partners or an overburdening of teachers, who are constantly visited by different partners. One teacher noted that the school has tried to limit duplication by combining project activities when possible. However, this overburdens teachers and is not always successful because the materials they receive from each partner include duplicated content. Stakeholders reported this communication and duplication problem cut across education and health activities because many overlapped within the school and required the same stakeholders (teachers, head teachers) to participate. For us, we can describe the coordination from our perspective as not functioning because, in essence, we just see things happening and do not have a clue on what. We suppose things could be different if some agency/government departments involved could brief us on this (R1, R4). — Males and females, School staff, Balaka We would like all those partners to explain their roles and clearly define what they aim at, so in the end, we can have an idea what the real issue is all about, unlike having each organization coming on their own (R3). — Males and females, School staff, Balaka We don’t know if these [projects] coordinate because we just see these stakeholders bringing similar things, so we can’t really know about this (R5). I think coordination is not there, because when they come to this school, each NGO meets its specific mentor (R4). But, as a school, we have tried to limit duplication in any sense, so we have the program checked when it comes, we sit down and analyze it, if we think it can be combined with some other project we therefore do so. This saves time and it helps keep things interesting (R2). —Males and females, School staff, Balaka February 2018 | ASPIRE Performance Evaluation Report 58 5.3 EVALUATION QUESTION 3: COORDINATION WITH DISTRICT GOVERNMENT Government officials reported that ASPIRE is good at coordinating with district offices by ensuring clear communication, facilitating training, and supporting supervision of interventions. Community stakeholders saw this collaboration through an increased presence of government officials at schools and in the community as facilitated by ASPIRE. When asked about how ASPIRE coordinates with district government bodies to reach its objectives, a number of government officials mentioned that ASPIRE has been good about sharing its plans with district offices. It seeks guidance before starting activities and holds workshops with the Ministry of Health, district education managers, and other stakeholders. Whenever they are planning their activities, they always ask for guidance from government offices. Whatever they have done so far, first of all they had to come to the Education District Office to inform us about their activities, and we gave the advice. They involved all the key stakeholders, and we could meet in various workshops with people from [the Ministry of Health], DEM and so many others. — Central government official Over the last year, ASPIRE reported presenting implementation progress and engaging district offices to plan activities in all three districts. It also met with District Education Network and district education manager in Balaka to identify opportunities for transfers for readmitted secondary school students, and organized a meeting with the Zomba DEC to present findings from the rapid needs assessment that identified schools with dire needs for latrine construction. Respondents noted the practical benefit of this level of collaboration. As one government official explained, such coordination ensures more effective support of ASPIRE’s goals and even helps improve coordination among the different government organs (e.g., MIE, the Ministry of Health, and the Directorate of Inspection and Advisory Services) by improving understanding of the distinct roles each one plays. He emphasized that, “just by just doing that, it helps coordination because each office understands the roles to support any project.” Community stakeholders commonly noted that they expected ASPIRE to coordinate with the district office. A group of school trainers commented that, “[the] District Education Manager is the entry point of each and every project, which has to do with education … the project must go and visit the District Education Manager first.” However, many saw this collaboration manifested through the increased presence of district government officials in schools and the community. ASPIRE staff reported that government officials have attended ASPIRE training, been engaged in training-of-trainers on ASPIRE’s literacy coaching and mentoring tools, and worked together to coach and mentor teachers on effective delivery of the Life Skills Curriculum. District field workers were engaged in supervision and monitoring school- and community-based interventions. Community stakeholders reported observing ASPIRE staff and government officials at schools and in the community together. February 2018 | ASPIRE Performance Evaluation Report 59 We gladly receive and implement activities by ASPIRE knowing they have a proven working record with government, especially from health and ours (education), so on this it appears ASPIRE is doing well. We have also seen the DEM, DEC, DHO [District Health Office], DC [District Council] officials participating in ASPIRE meetings and deliberations, and these are also acknowledged in handouts. —Male, School staff, Balaka Some noted a greater presence of government officials in schools, particularly by health workers, indicating (for them) an increased level of collaboration between the MoEST and the Ministry of Health. During school health days, ASPIRE reported supporting healthcare providers in the provision of services (e.g., HIV testing and contraceptive services), and strengthening collaboration with social service providers (e.g., police and child protection workers). Particularly for school staff, these activities were clear indications of increased coordination with district government bodies that did not exist before ASPIRE. We are working well with government officials, for example, government medical practitioners work and train with ASPIRE. There are referral agents from both the schools and the hospitals in which if we have a child who needs special medical help, these agents will write a form for the child to take to the hospital that will ensure that he gets quick and effective medical assistance. Should a child be abused, these agents will write an accompanying letter as to help the child get effective assistance at the police unit. —Female, School staff, Machinga Government ministries: health and education—during trainings, there is a combination of health workers [HSAs] and teachers. HSAs who were trained by ASPIRE handle sanitation issues, health talks, measles; there is communication between the two ministries. —Male, School staff, Machinga Respondents felt that ASPIRE’s coordination with districts could be further strengthened through increased monitoring, additional forms of communication and joint planning, and capacity-strengthening support to district structures. As school staff observed ASPIRE and government officials implementing activities together, they noticed that communication could be a problem. Community members were not always informed of project activities in a timely manner, or it seemed that multiple partners were conducting the same activities without talking to one another. For some respondents, this indicated a need to improve coordination and communication between ASPIRE and district government offices. A district health official in Zomba noted that there has been a struggle to coordinate partners in his district, which had led to some confusion in the DEC. In fact, several respondents from Zomba confirmed that coordination remained a challenge. A group of health services providers complained that ASPIRE did not adequately involve the District Health Office. Unfortunately, they miss out on working with existing district structures that involve partners – working group. ASPIRE does not work with any and so its visibility at district level is low … ASPIRE’s approach is ad hoc, which affects other plans. Different treatment on reimbursements; ineffective communication. For example, recently they had an orientation to their referral forms. Few February 2018 | ASPIRE Performance Evaluation Report 60 people attended because of poor communication. They need to improve. —Female, Community group, Machinga From ASPIRE staff’s perspective, they have noted a conflict between the education and health sectors in Zomba. A USAID staff member mentioned, “ASPIRE even had to stop their school health days because, not the Ministry of Health, but the MoEST said so … they worked through that and made some compromises with the districts to resume their activities.” An ASPIRE staff member affirmed that this type of struggle with coordination is the result of the conflicting policies of the Ministry of Health and the MoEST. In health policy, children can access contraceptives… In education policy, the policy is NO contraceptives are able to be accessed in school. You cannot even bring a condom to show the children to say, “this is a condom” when doing training and education. They are also not able to access contraceptives in the schools. —ASPIRE staff Many stakeholders recommended strengthening district structures to ensure better coordination at the local level. For example, the group of Youth-Friendly Health Service providers in Zomba recommended that ASPIRE improve its work through district coordinating structures to share objectives and work with partners. Alternately, a teacher in Machinga commented that what is missing is an umbrella body that oversees the implementation of these programs. Strengthening the District Education network and other existing structures. Teachers are overwhelmed with partners who report to schools at any time … —District government official We can enhance coordination by conducting different meetings so that we can know each other and do inform each other on what we do, as a result our coordination will be enhanced. —District government official In addition, school staff see monitoring as a way ASPIRE and district officials can further work together, and ensure that resources are being used to meet objectives. Although ASPIRE reported working with district offices to supervise and monitor school- and community-based activities, respondents were unaware of such activities and wanted more follow-up. The only opportunity for improving coordination is monitoring, to ensure the resources being given met their objectives. —Male, School staff, Balaka 5.4 EVALUATION QUESTION 4: ENGAGEMENT WITH PRIVATE SECTOR ASPIRE is on track to achieve or exceed its targets for private sector engagement, mostly through engagement with media partners. Perceptions regarding the possibility for further private sector engagement were mixed. Based on the FY 2017 Quarter 2 report, ASPIRE had already exceeded its target for the number of initiatives that should engage private sector actors for the year. Meanwhile, ASPIRE was well past the February 2018 | ASPIRE Performance Evaluation Report 61 halfway point toward achieving the target for the total monetary value of private sector contributions to be leveraged in support of ASPIRE’s goals, despite only being midway through the year. As can be seen in Exhibit 43, ASPIRE has surpassed its target in Machinga, and is over half way to meeting its target in Balaka and Zomba. Additionally, ASPIRE noted a donation of 12,000 books in 2016 as well as additional media contributions, although it did not assign a monetary value to these contributions. ASPIRE further reported that three of their four targeted initiatives supported by private sector are already underway. Exhibit 43: Resources ASPIRE has generated from private sector engagement as of FY 2017 Quarter 211 Performance Indicator District 2017 Targets FY2017 Amount of income and number of other resources generated from the private sector in support of ASPIRE activity (indicator 11) Balaka 3,855,000 5,000,000 Machinga 6,255,000 5,000,000 Zomba 3,855,000 6,500,000 Total 13,965,000 16,500,000 Source: FY 2017 Quarter 2 Report. Exhibit 44: Private sector engagement as of FY 2017, Quarter 2 Performance Indicator District 2016 2017 Targets FY2017 Number of initiatives supported by private sector stakeholders (indicator 53) Balaka 1 1 1 Machinga 2 2 1 Zomba 1 1 1 Total 4 4 3 Source: FY 2016 Annual Report and FY 2017 Quarter 2 Report. ASPIRE has engaged the primary sector in two main ways: (1) obtaining supplementary reading materials that support literacy instruction, and (2) engaging with media partners to air ASPIRE programming. The latter (ASPIRE’s work with the media) was the most well recognized type of engagement among ASPIRE and USAID staff. ASPIRE has signed two memoranda of understanding with Angaliba Television and the Malawi Broadcasting Corporation. They have also worked with Radio Maria and Luntha TV to air radio programs in recording and broadcasting ASPIRE activities and success stories. As far back as ASPIRE’s 2015–2016 annual report, the activity has mentioned plans to develop partnerships with print media houses, private libraries, and book sellers to make subsidized copies of resources available to schools. However, as of April 2017, this activity was still reported as “in 11 ASPIRE reported that calculations of income raised originated from support (among others) in form of reading materials and support by radio stations to air programs on the ASPIRE activity. February 2018 | ASPIRE Performance Evaluation Report 62 progress” and there is no mention of progress in this area (FY 2017 Quarter 2). The only mention of work with the private sector to procure resources was in regards to donations of books for ASPIRE activities made by a private benefactor at the local level. In line with the scope and form of these public-private partnerships, school- and community-level respondents were not aware of ASPIRE’s private sector engagement, but project-level actors, including USAID, ASPIRE, and consortium partner staff, reported greater familiarity. Among these stakeholders, challenges were identified in engaging with the private sector. ASPIRE and consortium partners pointed to Malawi’s weak private sector, which is often uninterested in collaboration with NGOs, as one challenge in meeting this objective. This has been a big challenge for us, and I think for Malawi, generally. Opportunities are difficult when the private sector is weak in a country. —ASPIRE staff Private sector is more interested in profit making and where they see nothing coming as profit, they don’t come forward to support activities. — NGO/Development partner In addition, staff noted that other partners, who had a more intentional focus built into their project design, were already covering the areas where ASPIRE would collaborate with the private sector. At the beginning of the project, we met with EGRA and reviewed EGRA’s Alliance Assessment Report completed by O’Brien and Associates International, Inc. … It didn’t seem to make sense to duplicate assessment efforts. I think it is important to note that the EGRA project had a dedicated, budgeted partner for this piece of work. I would question if project budgets normally have enough funding for the LOE [level of effort] it takes to seek out and develop local PPP [public-private partnership]—and if this is a strategic investment of project staff time, given other priorities. —ASPIRE staff Given that ASPIRE appears to be meeting its targets for private sector engagement, it is unclear whether the attention these respondents gave to challenges stemmed from the difficulty involved in achieving these targets, or an acknowledgment that the size of these targets makes them a relatively minor focus within ASPIRE’s scope, even while ambitious within the context of the Malawian economy and districts where ASPIRE works. At the district level, government officials and school staff highlighted the importance of radio broadcasts as an example of successful engagement with the private sector. However, these respondents highlighted PSI’s Youth Empowerment radio program rather than ASPIRE’s programs, and were not aware whether PSI was collaborating with ASPIRE in this regard; neither ASPIRE reports nor interviews indicated that ASPIRE is collaborating with PSI in this regard, although it could be an area of focus for the future. In some schools, PSI has provided radios for Youth Alert radio program in Radio Listening Clubs. The partner I know working here is Chinasi foundation that is concerned with education of males and females, but I am not sure if ASPIRE works together with them. —District government official PSI Malawi, it has introduced a radio program to improve access to information, especially on SRH [sexual and reproductive health], through the youth alert mix program, through Pamawa ndi February 2018 | ASPIRE Performance Evaluation Report 63 achinyamata project and Save the Children that deals with WASH (R2). The program started in January this year, and our leaners listen to this program on Saturdays, and after which we have discussions. —Males and females, School staff, Balaka The access to health messages by the girls through radio programs has given the girls options on their health and sexualities, and overall there is a reduction in drop outs due to pregnancies, though we cannot attribute that only to ASPIRE. — Males and females, School staff, Balaka 5.5 EVALUATION QUESTION 5: ACCOMPLISHMENTS, BEST PRACTICES, AND LESSONS LEARNED Respondents singled out ASPIRE’s hygiene and sanitation activities as a key accomplishment, pointing to the construction of changing rooms and distribution of sanitary pads, and improved hygiene that resulted from these. As reported under Finding 7, Finding 8, and Finding 9, ASPIRE’s hygiene and sanitation activities were reported as important influences in ASPIRE’s progress toward the overall development objective and Outputs 2 and 3. In addition, when asked about ASPIRE’s most important accomplishments and best practices overall, school staff and community groups highlighted ASPIRE’s activities in girls’ hygiene and sanitation. Community members cited the important role of mothers’ groups in teaching girls about menstrual hygiene and why they should stay in school during their menstrual cycles, as well as their role in making and selling sanitary pads for the girls. I am very happy with the changing rooms we have now and because of this, girls are no longer staying at home [during the] monthly period. —Female, Community group, Zomba There is progress, especially in sanitary pads; girls accepted this thing and really, they are being helped with it, because it was difficult for them to come to school while menstruating. In the past, before ASPIRE, a child could stay 3 days without coming to school because of [her] monthly period. —Male, Community group, Zomba ASPIRE has taught us a lot about education and hygiene. Through this project, we can point out at notable changes in terms of hygiene and sanitation and an improvement in girls’ education. —Male, Community group, Balaka Teachers, head teachers, and PTA members highlighted that the reason ASPIRE’s hygiene activities were best practices was their sustainable nature. Respondents noted that even if ASPIRE ends, they now have facilities for girls to use at school when they are menstruating. They also reported that because mothers’ groups are producing the sanitary pads locally and using it as a business model, this practice could be continued well into the future. Some teachers reported that other schools are replicating the changing rooms at their schools. The issue of sustainability and sanitation, it really impressed me. On sustainability we now know the skills we can use to bring a child back in school and even if ASPIRE project comes to an end, we will February 2018 | ASPIRE Performance Evaluation Report 64 continue. On sanitation, we now know that in every school we should have buckets for the students to use it after using the toilet and before eating. —Male, School staff, Zomba The issue of menstrual hygiene, the toilets built by ASPIRE is something worth noting. Last week, I had two head teachers form Madziamela and from PEA [primary education advisor] to see how the room was constructed and they want to construct using their own resources, which means that the locals have welcomed it. —Male, School staff, Machinga The mother groups have learnt to sew menstrual pads, which have contributed great change in school attendance of girls … They make very good pads, which are sold, then a small percentage goes to Scotland [NGO supporting this activity] to buy more materials. —Male, Community Group, Machinga ASPIRE’s community engagement to influence attitudes and encourage collaboration among all stakeholders has led to increased knowledge of health, more support for girls and boys to attend schools, and, ultimately, greater demand for girls’ health and education. USAID and ASPIRE staff highlighted ASPIRE’s accomplishments in creating community demand for girls’ improved health and education. Their work with a variety of community stakeholders included mothers’ groups, PTAs, school management committees, and traditional leaders. Combining this community-based work with activities targeting school-based staff and healthcare workers was reported as a comprehensive approach that should be seen as a best practice. This kind of mobilization was helping the community to have their own cultural practices. Sometimes they weren’t thinking about how they were impinging on the rights of the girls, and you know, “we weren’t seeing them negatively, but now we’re looking at them in new light.” And this has been the big thing. We’ve been working with traditional leaders and village initiation counselors and looking at those practices. Sometimes they weren’t looking at the time it was taking, and now they’re looking at the school calendar. —ASPIRE staff They are good at mobilization. Mobilizing communities, getting them to participate. To mobilize community members to support activities. Relationships they have built with the community members. —USAID staff So, the demand creation they’ve been able to do among really young students, boys and girls, to actually feel comfortable enough to come and access services, I think that’s one of their greatest successes. I get really excited each quarter to hear about school health days, and it’s not just a referral, it’s actually an uptake of services by students, and that’s something I think really special to this program. —USAID staff School-level staff, community groups, and government officials echoed the importance of this demand creation and highlighted their own increased knowledge of the importance of girls’ education and health. The increased demand was reported to be a result of ASPIRE’s work in February 2018 | ASPIRE Performance Evaluation Report 65 expanding understanding of the importance of girls’ education through ASPIRE teacher training, and improved communication skills regarding girls’ health among community groups. I have learnt a lot with ASPIRE project. I have learnt to treasure the girl students and how best I can also raise my girl child. I have also gained communication skills and how I can effectively communicate. —Male, Community group, Zomba I have learnt on how I can work with different people, and also I have learnt on how I can talk to drop out girls so that they can go back to school because ASPIRE has given us power to go around the village and talk to the communities on how we improve girls’ education. —Male, Community group, Machinga Community groups and government actors reported feeling empowered by being able support their community, as a result of coordinating and working with ASPIRE. Coordination, we have learnt to come together and work as one, not all of us are from the same village, but we are working together and achieving results making an impact. That briefing stakeholders should happen quite early in the program. —Females, Community group, Balaka Respondents from all stakeholder groups reported the engagement of traditional leaders and other community leaders as a key part of this community engagement approach. The chief has employed a group of individuals who move around households to find pupils who are not going to school and grab them to the chief who summons their parents. (R1) Part-time classes in our communities have helped us in improving our reading skills (R2). —Males, Students, Machinga Government officials and students reported that ASPIRE’s work with the community has played an important role in increasing students’ awareness of the importance of their education and health. Firstly, I am impressed by this project that has come with ASPIRE, called girls’ empowerment. Now, girls are able to exercise their freedoms, as in participate freely in different activities such as competitions, unlike in the past, especially during my time, when we left all the things to be done by the boys. Now the mindset of people has changed that a girl is not a giving-birth machine. —District government official Although the majority of students reported community members’ being supportive and encouraging school attendance in support of ASPIRE’s main objectives and goals, some students shared having varied experiences, expressing that some community members’ attitudes hindered or were indifferent to girls’ and boys’ education. Students mentioned being sent on errands when they were reading or studying; community members’ not knowing the importance of school, only helping their own children out of jealousy, or listening to loud music; and expectations for attending funerals or other gatherings. The continued presence of these barriers highlights that although ASPIRE’s community engagement has helped change attitudes, children still continue to experience barriers to their health and education. February 2018 | ASPIRE Performance Evaluation Report 66 The importance of training was a key theme among beneficiaries when asked about accomplishments, lessons learned, and best practices; those receiving training and other support felt that this capacity building, particularly for teachers and mothers’ groups, was an important accomplishment. Respondents stated that teachers and community actors trained by ASPIRE, particularly mothers’ groups, are a key influence on students’ knowledge of sexual and reproductive health, helping raise their awareness. While school and ASPIRE staff highlighted accomplishments in sensitizing community stakeholders and achieving collaboration across actors, government workers focused on training of teachers and mothers’ groups as the key factors facilitating the effective cross-sectoral approach. Yes, there it has done pretty well through mothers’ groups, who were training to follow up on boys and girls who dropped out of school (R1); teachers were taught on positive attitude[s] towards girls and boys who have agreed to come back to school (R4 and R5). —Males and females, School staff, Machinga Respondents across different stakeholder groups emphasized the importance of working with mothers’ groups. Mothers’ group training held by ASPIRE was believed to be successful in general, and respondents specifically highlighted the importance of their role in keeping girls in school. Mother groups are key to keeping girls in school … They help us in so many ways. Sometimes they gather all the girls and counsel them that “when you see such a thing, this is what you must do.” So, they help us a lot. They come several times to counsel the girls during the term. —Female, Community group, Machinga Mother[s’] groups are playing a very good role in encouraging a girl child to go back to school. —Male, School staff, Zomba Qualitative data from the community indicate that ASPIRE activities have influenced students’ level of knowledge, mainly through working with teachers and mothers’ groups. All respondent groups lauded mothers’ groups for their role in educating girls on sexual and reproductive health, bringing girls back to school and sensitizing the community to the importance of girls’ education. During focus group discussions, female and male students most often reported mothers’ groups trained by ASPIRE as having the biggest influence on increasing their knowledge of sexual and reproductive health. Girls and community members also referenced the important influence of female role models. [The mothers’ group] encouraged me to work hard and abstain to avoid unwanted pregnancies. It also advised the using of family planning, like condom, in case we cannot manage to abstain. —Female, Students, Balaka Mother[s’] group[s] do advise us that if we want to have sex we should be using condoms and we should not have different affairs. —Male, Students, Zomba February 2018 | ASPIRE Performance Evaluation Report 67 As of now we don’t see that [early pregnancy] happening, because sometimes ASPIRE do come with role models so that the girls should be motivated and work hard in their studies. —Male, School staff, Zomba Community groups, teachers, and girls also highlighted the importance of ASPIRE’s Life Skills training for teachers in improving girls’ HIV knowledge. In secondary schools, ASPIRE supported the purchase of approved Life Skills Curriculum textbooks to ensure that schools had enough copies. In addition, ASPIRE developed continuing development manuals to provide teachers with instructional support, and sample teaching and learning strategies for the existing Life Skills Curriculum. ASPIRE piloted this training and then, after assessing teachers’ use of the curriculum, adapted it and added a continuing professional development module to focus on areas that teachers felt uncomfortable teaching. Teachers not only incorporated this curriculum into their lessons, but also expanded their teaching to school health days—also identified as an important influence on girls’ education. I know of a time ASPIRE came to this school, and we invited all the girls to secondary school hall. During this meeting, the pupils were told about openness concerning their HIV status and there was also a health surveillance assistant who talked a lot about nutrition with the girls. —Males and females, School staff, Zomba My madam taught me about menstrual hygiene (R2, R3, R6) and encourages us to go for VCT [voluntary counseling and testing] (R5). —Females, Students, Machinga We learn that in class when we are learning life skills, and our teachers encourage us to go to VCT to test our blood so that we can know our status. They even encourage us to be clean when we are doing our monthly period (R2, R3). —Females, Students, Zomba These qualitative data align with 2017 KAP data, which show that the influence of teachers and community members in promoting HIV/AIDS awareness is pronounced, as seen in Exhibit 45. KAP data identified teachers as students’ main source of information in learning about HIV/AIDS (81 percent in primary and 86 percent in secondary), followed by hospitals, relatives, and friends. February 2018 | ASPIRE Performance Evaluation Report 68 Exhibit 45: Where students indicated having heard about HIV/AIDS 34% 31% 30% 36% 71% 69% 32% 37% 27% 42% 49% 49% 41% 39% 85% 85% 35% 33% 39% 43% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Male Female Hospital Radio Teacher Friends Relatives Hospital Radio Teacher Friends Relatives Primary (n=580)* Secondary (n=381)* Source: 2017 KAP Survey Questionnaire; data available in Table 13 (a and b). * 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (see Exhibit 30). Community groups and teachers also reported the importance of traditional leaders, such as village chiefs and initiation counselors, in increasing students’ knowledge of sexual and reproductive health. For example, initiation counselors are shifting to incorporate family planning into their girls’ education and the importance of safe circumcision in their boys’ education. “Introducing civic education to the communities” in order to end some cultural practices (R3), ASPIRE has not managed to change everything but it’s doing a good job; they taught village and religious leaders on the importance of [these issues] (R2). —Males and females, School staff, Machinga Government officials and school staff highlighted teacher training in literacy and life skills having a positive result on students as a key accomplishment of ASPIRE. Primary teachers reported that the greatest lessons learned were increased teaching skills in Chichewa, English, and life skills gained during the ASPIRE training. School staff also commented on the variety of training types, which enabled teachers to improve their skills in teaching language, reading fluency, word recognition, comprehension, and life skills, as an important accomplishment of ASPIRE. It was in 2015 when I was teaching English in Standard 5 and I was applying the skills I learnt from ASPIRE … I asked the leaners that they will only go out after they read a passage and to my surprise almost all the learners managed to read and I was very happy that ASPIRE has done a great job. — Male, School staff, Balaka February 2018 | ASPIRE Performance Evaluation Report 69 Teaching using practical examples make[s] learning fun. I have also learnt that not mixing English and Chichewa helps learners to grasp English language better. Before ASPIRE, we used to mix English and Chichewa but ASPIRE says if it’s English lesson, only use English and use illustrations to explain difficult words. The same when it is Chichewa, do the same and I have seen this to be very helpful for the learners. The training we had with ASPIRE was very good. —Female, School staff, Zomba In the past when we want to find meaning of a word, we always went to the dictionary to find it, but now we have a technique where we find the meaning through relations of the context. We have students now knowing how to read and write. There are strides made with the speaking part, but now students can answer questions easily and find meanings of new vocabulary easily. As teachers, we have new various techniques too, from planning our lessons to delivery of the lessons. This has made our work enjoyable and easier than before ASPIRE came (R2, R4, R1, R5, R6). —Males and females, School staff, Balaka Successful integration with other USAID partners was reported as a key ASPIRE accomplishment, but it has not been easy; this has led to several lessons learned. ASPIRE development partners had many successes and challenges in integrating their activities. Project-level stakeholders (USAID and ASPIRE staff) mentioned that ASPIRE’s cross-sectoral integration should be replicated by other partners within and outside of the education and health sectors to provide beneficiaries with a comprehensive and holistic offering of services. Opportunities to improve on this model from the perspective of other partners can be found in Finding 13. Project is integrated and mission is promoting integration even in activities that aren’t integrated. Even in the district in different USAID activities in health, agriculture, etc. ASPIRE has been used as a model to encourage and promote that integration even where we don’t have a program that is integrated, but the District has different aspects of USAID funding and support. —USAID staff It has worked because I have seen somewhere where they had health days and somebody come in to provide education and someone come in to provide services so because most of the things is not integrated, someone will give you information on where to get tested, health issues, etc. and then the child is given where to get that but they don’t get that because the services are not close by. What I see the integration is when they have open [health] days, any of the issues, the learner is given the information on how to do that and they can access this service. —USAID staff 5.6 EVALUATION QUESTION 6: BENEFICIARIES’ RECOMMENDATIONS ON ADAPTATIONS While respondents across stakeholder groups praised ASPIRE’s cross-sectoral design in terms of its holistic approach, they also pointed to the substantial management challenges this model creates in implementation. Stakeholders at all levels recommended improving monitoring, and project-level actors pointed February 2018 | ASPIRE Performance Evaluation Report 70 to the need to streamline management across the sectors to strengthen the dividends of the activity design. Across stakeholder groups, respondents were concerned about the lack of monitoring in ASPIRE activities. Teachers complained that although ASPIRE does a lot of good training, it does not follow up to ensure that training content is implemented correctly. School staff and community groups actively requested feedback on the activities they were conducting with ASPIRE support. ASPIRE or any of its implementing partners, such as FAWEMA, should monitor us and see if the things they have taught us are being implemented well. We feel this can encourage us and make us to be more serious with our work. —Females, Community group, Zomba We like it when MIE supervisors visit us to observe the teaching, but the problem is that they do not give us feedback nor make corrections where we have made mistakes for improvement unlike when the PEA [primary education advisor] visits. He, after observing, gives us feedback and corrective measures. —Males and females, School staff, Machinga The only thing that seem[s] to be lacking is lack of monitoring, it appears that teachers have been trained, but no follow ups have been done to track performance changes. —Male, School staff, Balaka Government officials echoed this weakness in monitoring, and identified monitoring as a key activity to ensure that ASPIRE activities are effective. A government official in Machinga noted that ASPIRE should “strengthen the monitoring aspect of the project, which is weak.” Project-level actors agreed that monitoring has been limited in the past 2 years, but is necessary to secure the progress to date. For education and literacy, we need to intensify monitoring and support. This involves literacy specialist and project officers visiting schools to observe literacy lessons and help to mentor and support the teachers as they deliver lessons. —ASPIRE development partner From project-level respondents’ perspective, the integrated, cross-sectoral model has made the activity simpler to implement. Beneficiaries do not put themselves in sector silos; for them, the comprehensive nature of the model makes sense. However, challenges have emerged in managing standards and merging indicators across health and education, resulting in cross-sectoral tensions. Monitoring requirements have remained disjointed and internal mechanisms have not helped improve processes to deliver, track, and follow up on activities. Quality: The designs are good, the structures are good. However, the processes and procedures—they need to improve on their monitoring. —USAID staff The integrated model requires additional reporting requirements that would not exist with a single￾sector project. With the rollout of DREAMS, efforts were made to harmonize activities and monitoring with the education office, but ultimately, since ASPIRE has four different funding streams, it is accountable to four different USAID teams, each with its own agenda. Several ASPIRE staff members commented on the struggle to reconcile the perspectives and assumptions of the health and education sectors. This was exacerbated by introduction of DREAMS, which had a specific agenda for its work. February 2018 | ASPIRE Performance Evaluation Report 71 But there is room for improvement. Internally, as USAID has different offices that are supporting ASPIRE project. We do not frequently sit down and look at ASPIRE as one program that has got one goal. We still work in silos. —USAID staff In addition, USAID guidance changed ASPIRE indicators significantly mid-implementation, further complicating monitoring activities. One USAID staff member reflected that a key lesson learned is to be clear about how the sectors work together, as well as how they are held responsible for each result area. This has had an impact on implementation and monitoring of key activities. The integrated model came with a lot of additional reporting requirements, which even up to this day we’re struggling very much with ASPIRE on. —USAID staff USAID staff reported that although the integrated model made sense at the beneficiary level, the challenge was in reporting to different USAID offices with different indicators and different expectations for their projects. ASPIRE staff reported similar challenges with reporting to different offices and managing expectations for standards across the health and education offices. Additionally, because funding streams operate on separate timelines, activities cannot always be conducted in the desired timeframe, a fact that complicates workplan management. These challenges were compounded by changes to the intervention districts and content in 2016, which required substantial management oversight. One of the lessons—and maybe it’s not as much ASPIRE as the Mission—is to be very clear about how the projects work together as a whole, and yet still being accountable for each of the different results areas. —USAID staff I think what we learned is that while integration is great, we did at some point quite overwhelm ASPIRE. They were doing EVERYTHING. So they’re really, they ARE experts in education, and here we were asking them to be experts in HIV prevention. So, we learned as a Mission, the key is for ourselves, as we go forward with integrated projects, if you’re stretching them beyond the area of their expertise to ensure that the technical assistance is really there. —USAID staff Stakeholders at all levels want better harmonization and communication among USAID partners, and between partners and school- and community-level actors. They recommended strengthening and making better use of district structures, and leveraging partner resources to better integrate activities. Respondents at all levels noted that activities needed to improve harmonization, both between ASPIRE partners, and between beneficiaries and partners. Local stakeholders did not distinguish between USAID-funded partners or other NGOs doing similar work; they observed duplication of efforts by NGOs and a lack of communication with school- and community-level beneficiaries regarding these different efforts. Project-level actors were also aware of these issues, as one respondent acknowledged, “Sometimes we confuse the stakeholders because they see a lot of duplication of effort and a waste of valuable time.” Respondents saw the potential to merge the efforts of different actors and conduct activities together. February 2018 | ASPIRE Performance Evaluation Report 72 There is need for organizations to coordinate activities at a higher level, and if possible partner in conducting similar activities. This has an advantage of deriving vibrant lessons from different approaches. However, the challenge remains, which is fundamentally [that] all organizations have different goals, objectives and methods and each has to abide by theirs. —Male, School staff, Balaka Government officials, NGO/development partners, and project-level actors recommended increasing work through district structures as a practical means to achieve better alignment across the different actors. Respondents specifically called for aligning objectives and activities, monitoring progress, and learning from one another. We should all be operating from Teacher Development Center and the PEA [primary education advisor] should be key in all these meetings. We need to share best practices and not competing. — USAID development partner Several respondents suggested better use of the District Education Meetings—by all partners—to submit updates and harmonize efforts, but this request is not unique to ASPIRE. This suggestion was independently provided by a community group, ASPIRE implementing partner, and a district official. The most important opportunity missed by all partners (at district level) is that instead of using the District Education Meetings for planning that is not done. Each partner seems to compete for no good reason at all. As you are aware, all partners and projects are members of [DEC] meetings. Only if these DEC meetings are properly used for example as joint planning meeting, annual and quarterly plans, partners would be able to identify sister partners/projects and collaborate in advance on how to work together and achieve the intended goals. —ASPIRE Development partner The District Executive Committee should be the entry point in this way coordination could improve. —Male, Community Group, Zomba All respondents wanted continued and increased engagement with the community through awareness-raising campaigns, engaging chiefs to address bylaws, working with parents, motivating local volunteers, and supporting and training mothers’ groups. During interviews and focus groups discussions, respondents made it clear that ASPIRE should continue its varied community-engagement activities. Community groups and school staff reported that the sensitization of the community on the importance of girls’ education would be sustained even when the activity ends. However, more effort could be made to build awareness in communities. A few respondents suggested providing more opportunities for schools to meet with parents and other community members. We should all work together we teachers, parents, mother groups—in short, the whole community, we should make sure that we keep our children in school and also abolish all bad cultural barriers such as forcing girls into early marriages. —Female, School staff, Zomba February 2018 | ASPIRE Performance Evaluation Report 73 Chiefs continued to be identified as key figures in the community who can reduce or eliminate harmful cultural practices by implementing bylaws to pressure parents. I feel what ASPIRE should work with chiefs in raising awareness about sexual and reproductive issues and make sure there is a penalty to parents who encourages sexual practices and early marriages. There should also be support given to the learners who are re-admitted to school after childbirth. —Female, School staff, Balaka Several school staff members mentioned that chiefs need to be involved in requiring parents to send their children to school. While there is compulsory primary education, teachers in Zomba noted there is no enforcement, “We still see children roaming around at 6 to 7 a.m. and their parents are just looking at them.” A group of head teachers in Machinga made the following recommendations: [First], some authorities should encourage our chiefs to establish bylaws on parents who allow their children to get marriage, or who organize marriage for their children with older men; [second], need to have stiff punishments for parents who infringe the rights of their children; [third], abolish bad cultural practices such as chinamwali because these encourage early sex among boys and girls. —Male, School staff, Machinga As discussed in Finding 24, all respondents saw training activities as a key capacity-building activity. For community members, however, additional resources were sometimes needed to implement the knowledge and skills gained from training. Stakeholders wanted ASPIRE to consider providing incentives or additional resources for volunteers (e.g., mothers’ groups, health surveillance assistants, and tutors) to implement activities in schools and the community. There should be provision of some incentives, especially to the volunteers like mother[s’] groups and [health surveillance assistance]. For instance, bicycles for transports or allowances. —USAID Development partner ASPIRE should at least support the mother[s’] group with [a] sewing machine so that they can be making sanitary pads for the girls. —Female, School staff, Zomba I would suggest that after training, the chiefs should be given bicycles to follow up [with] children who are not going to school. They can go around the video shows to trace children not attending school. They can give T-shirts to those making follow-ups. —Male, School staff, Machinga There is overwhelming support for ASPIRE’s training for teachers, government officials, and community members, with an emphasis on involving more participants and conducting refresher training for those who have already participated to ensure sustainability of activities. ASPIRE training participants overwhelmingly saw the training and skills they gained as something that would be sustained beyond the end of the activity. However, it is clear that respondents wanted more training and wanted more individuals to receive training to solidify those gains; the vast majority of these references were connected to training conducted in primary schools. Government February 2018 | ASPIRE Performance Evaluation Report 74 officials and mostly primary-level school staff emphasized the need for refresher training for teachers, as well as more training for all teachers, particularly considering the possibility of transfers and loss of institutional knowledge. A number complained that previous training activities had been far too short to adequately cover the content and requested refresher training. In general, however, respondents did not offer specifics on the content they wanted. In some cases, the need for more training was connected to the fact that there are still challenges in teaching and learning. When respondents did specify the topic, it did not necessarily relate to ASPIRE’s intervention areas, as in the case of a number of teachers who mentioned the need for math and science training. One school trainer specifically wanted retraining on handling of large class sizes, which she noted was still a struggle for teachers, despite the ASPIRE training. We have noted again that the teachers still have problems regarding to the things they have been trained. I would propose that they have refresher trainings. And I feel that the teachers have not been trained enough because they only had small sessions of trainings, so more trainings should be done. — District government official ASPIRE should continue training more teachers whilst providing refresher sessions for the already trained in different skills on how we can deliver our lessons for the leaners to understand well. — Female, School staff, Zomba One school trainer thought ASPIRE should lobby for the inclusion of reading instruction coursework in the Teacher Training College syllabus, so that incoming teachers would have the skills they needed and to ensure broad coverage. ASPIRE should lobby inclusion of reading skills in Teachers Training College syllabus so that the time they qualify they have the skills. Rolling out may still not ensure that every school has relevant skills. One thing projects forget is that we are civil servants and so get transferred to other districts leaving a gap. —Female, School staff, Balaka Government officials also supported integrating this material in Teacher Training College education; responses indicated a desire to see this training expanded to other districts. Community groups and mothers’ groups echoed the need for more frequent training to help them gain expertise in what they do. In general, these responses indicated that respondents want either refresher training to solidify their learning or new training to build additional skills. For example, at least one mothers’ group had not received training on how to make sanitary pads, but felt it was needed. Stakeholders reinforced the idea that more training could help sustain ASPIRE’s activities. A group of teachers at a primary school emphasized distribution of training manuals to all teachers, whether trained specifically by ASPIRE or not. ASPIRE needs to produce manuals to every trainee, so there can be sustainability of the impact, so even after ASPIRE pulls out the skills can be left behind; also continue reaching communities and training mother groups for repetition, [which] re-enforces the change that has already been made; also, February 2018 | ASPIRE Performance Evaluation Report 75 it should be clearly announced to the stakeholders the roles as an exit strategy (R1, R5, R3). —Males and females, School staff, Balaka In addition to including more teachers in training activities, a few respondents commented that more members of the community should be involved, so that there is broader buy-in. This was particularly in reference to chiefs, as discussed in Finding 23. When ASPIRE calls for training, they tell us to bring with us one or two chiefs, and we usually take the senior group. When he goes there, it appears that he gets paid. So, the other chiefs they do not embrace the concept fully to take it to the people because of these issues. We were given money by ASPIRE to train the chiefs but we did not train all of them. I would prefer a single training for all village heads surrounding the school so that they all get the same message. —Male, School staff, Machinga Local stakeholders and students want continued and additional direct contributions from ASPIRE (e.g., toilets, changing rooms, books, bursaries, and classroom blocks), but are concerned about sustainability. Community stakeholders, including school staff, mothers’ groups, community groups, and students, recommended that ASPIRE provide schools with more resources to meet its objectives, particularly around WASH structures and resources that directly support students’ attendance. Respondents saw direct contributions (e.g., toilets, classroom blocks, books, and bursaries) as key ASPIRE inputs that have made the activity successful. However, local stakeholders saw more needs that should be addressed. In terms of materials, respondents emphasized that schools needed books and desks, as well as a laboratory, library, and toilets. A few respondents noted that changing rooms and toilets had not been built as promised by ASPIRE. Others requested that ASPIRE extend its program of building these structures to all schools, noting that their schools had not benefited from structural improvements. They should have handled the serious limitation of resources; e.g., changing rooms were not constructed as promised by ASPIRE during the trainings. They promised to provide us with hand-washing facilities, which is something they did, but they fail to provide the school with soap. —Female, School staff, Balaka The issue of toilets will have to be addressed ... We should be assisted in building as many toilets as required. ASPIRE brought books, life skills books; this was greatly appreciated. So yes, we desperately need books, and every organization that arrives must try hard in assisting us on finding the books. The bursaries should be extended to include a greater number of girls … it would be of great importance to support the needy girls. —Female, School staff, Balaka An ASPIRE staff member acknowledged that they have not been able to meet the needs of all schools in term of WASH structures: There was just a tremendous need, so while what we’re doing is important and valuable, we didn’t have money to address water access issues. So, drilling boreholes is very expensive, and we didn’t have February 2018 | ASPIRE Performance Evaluation Report 76 the money to address the needs of schools … we are finding that we might be able to do a few, but we definitely didn’t have funding to do all the schools, and we had to prioritize. So, with that work, there’s a continuing gap, and we could only make so much headway toward that gap, because of the funding restrictions. —ASPIRE staff Community groups and mothers’ groups were particularly concerned about uniforms, which motivate girls to stay in school, as well as the provision of books and stationery. Several teachers and mothers’ groups commented that hostels were especially needed on secondary school campuses, noting that many students have to walk long distances and “are not protected out there.” We would also love if ASPIRE built some hostels at [our school]. Currently, girls would be selected to go at the secondary but get pregnant because it is far and are taken advantage of on the way to and from school. —Female, School staff, Balaka All respondents, but particularly students, emphasized the need for bursaries to both incentivize students and support poor families who would otherwise be unable to send their children to secondary school. Students overwhelmingly felt ASPIRE could help them through the provision of school fees because their parents could not afford them. There are a lot of challenges for girls, provision of bursaries to students, especially needy ones, will help most of us achieve our carrier goals, because there are a lot of bright students that quit school due to monetary issues. —Females, Students, Balaka Yes, there are such matters, for instance, girls that make it to secondary school; some are having problems to pay for their school fees, which may make them drop out of school. If ASPIRE can provide scholarships to these it would be welcome; this would even be an incentive for them to work hard (R2, R1, R5). —Males and females, School staff, Balaka However, community groups, school staff, and government officials reported that they did not see direct resource provision as sustainable. Most government officials reported that the direct resources ASPIRE provided in the form of school block grants would be difficult to sustain after the activity ends. A group of students noted that those currently supported by ASPIRE bursaries would likely drop out. Many girls and boys who are paid fees by ASPIRE will drop out of school because they were used to be beneficiaries hence will be confused and parents will fail to pay for their child as a result will just be staying at home (R3). —Males, Students, Machinga I am only worried about the sustainability of the project. There are some areas that might be sustainable but there are others that will be hard to sustain, an example will be the school feeding programs the schools have started under ASPIRE. We should find a way to empower schools. — District government official I hear ASPIRE had only 2 years to do its activities. I will give this example, ASPIRE is helping a form 1 student, and after 2 years when ASPIRE is leaving the student is in form four. What will happen to February 2018 | ASPIRE Performance Evaluation Report 77 this child? I think it will be a disaster. I think it will be better if this period was extended so that at least this child can benefit well. —Male and females, Community Group, Balaka Local stakeholders recommended that ASPIRE include boys, particularly in the provision of bursaries, to reduce competition and draw more support for girls’ education. School staff commonly mentioned that ASPIRE should include boys in all of its activities. Although boys are included in almost all of ASPIRE’s work—including school health days, reading interventions (teacher training and learning materials), the referral system, WASH infrastructure, and classroom Life Skills education—they are not included in after-school clubs, nor do they receive bursaries to the degree girls do. Their responses emphasized that boys’ exclusion is discouraging them and could lead to disparities in the long term. Some school staff members did not seem to appreciate the reasons for explicitly focusing on girls’ education, emphasizing issues of what they perceived as fairness and inferring that boys could not handle the disparity. We would like to ask that ASPIRE should diversify focus from girls only to include boys, because the scales might be tipped in future, having well-educated women and uneducated men… —Males and females, School staff, Balaka ASPIRE should also focus on boy child education, because in the end, if they don’t, the boys will not be empowered. —Male and females, Community group, Balaka The concentrating on girls only makes the boys feel bad, and as the result they end up having affairs with them; as a result, girl[s] are ending up getting pregnant. —Male, School staff, Balaka Other school staff members noted that schools have had to deal with an increased number of complaints from boys’ parents since ASPIRE started. These respondents noted that through the inclusion of boys, particularly in the distribution of bursaries, more support would be drawn to girls’ education from parents and students. I think inclusion of boys as secondary target will lead to total support from both the boys themselves and their parents. —Male, School staff, Zomba There was a general fear that as girls are being empowered, boys could be disempowered and wanted ASPIRE to strike a balance on this. —Males and females, School staff, Balaka Lack of inclusion of the boys in the project results in reduced support from parents not having girls. — Male, School staff, Zomba Youth of all ages want more extracurricular activities sponsored by ASPIRE, especially sexual and reproductive health-focused youth clubs and forums, as well as openness from parents and teachers regarding these issues. School staff recommended more role model programs for students. February 2018 | ASPIRE Performance Evaluation Report 78 Primary and secondary students alike want more activities and youth clubs dealing with sexual and reproductive health issues in their schools. Secondary students particularly considered youth clubs as a critical means to learn more about these issues. They recommended holding more forums and meetings, such as bringing health experts to answer questions and discuss these issues, and ensuring that parents push their children to attend. Punishing students who do not come to listen to the Youth Alert (R5). They should form a Youth Club so that we should be discussing the topic (R3). Organize inviting the health experts (hospital) to be coming to counsel us here as sometimes we are discouraged with the distance to go there (R3). There should be freedom of accessing family planning like condoms at the school (R5). —Females, Students, Machinga We would like the school to partner with health experts so that periodically we can interact and ask questions, and as for hygiene, we would like the school to consider constructing latrines and providing more trash bins (R4, R3). —Females, Students, Balaka Secondary school students wanted more resources generally, such as access to books on sexual and reproductive health, and condoms for those who are sexually active. It was clear that primary and secondary students want their parents and teachers to be more open with them about sexual and reproductive health issues. One girl commented that if parents and guardians are shy about the issues, then “they should send us to initiation ceremonies where we can learn more.” For us it would be helpful for our families to encourage us [to] engage in youth programs and associations like Youth Alert that we can learn more; we also wish our parents could be more open on these issues especially fathers (R5, R7, R4). —Females, Students, Balaka It would be helpful if our teachers could be more open on issues of sexuality, even during lessons like in Life Skills they should be flexible to explain in detail about [sexual and reproductive health] issues. The school should also involve health workers to counsel us often (R3, R1). —Females, Students, Balaka A number of school staff members discussed ways to support girls in schools, specifically through promoting role models and mentoring. A group of teachers in Balaka suggested career talks from role models from across the country to act as living examples and training community volunteers to help learners with homework outside of school. One teacher from Zomba suggested an entrepreneurship program to help girls work after graduating. ASPIRE should also facilitate training volunteers in the villages who are helping learners practice reading outside the school premises [and] facilitate career talks from different role models from different places across the country (R2, R6, R5). —Males and Females, School staff, Balaka A number of students commented that role models would be useful in helping them reach their goals, “so that I can dream big.” Several boys noted that although they want role models to come and encourage them, they do not know how to find them on their own, emphasizing that “role models should reach us.” February 2018 | ASPIRE Performance Evaluation Report 79 6. CONCLUSIONS AND RECOMMENDATIONS This section presents the evaluation’s conclusions and recommendations. The conclusions synthesize the evaluation findings with information from other available sources; they are organized by the design and results, with facilitators/enablers and inhibiting factors/constraints noted throughout. The recommendations are based on input received from ASPIRE, USAID, and activity stakeholders during a findings workshop with 35 participants held in Lilongwe on January 25, 2018, in conjunction with evaluation findings, best practices from relevant literature presented in the conclusions, and the perspectives of subject matter experts on the evaluation team. 6.1 CONCLUSIONS Across all three outputs (education, health, and structural/cultural barriers), stakeholders see ASPIRE’s activities as relevant and confirmed that the interventions are appropriate mechanisms to promote the desired change. This relevance is seen individually within each output and, more importantly, in the added value of ASPIRE addressing all three outputs holistically. Finding 4 illustrates that the teaching and learning materials, extracurricular activities, and continuing professional development program for teachers are helping build school capacity for high-quality literacy instruction at the school level and extending opportunities for students to develop an appreciation for reading. Finding 6 shows the potential of school health days and Youth-Friendly health services referrals to promote healthcare-seeking behaviors, while Finding 19 details that ASPIRE’s capacity-building work in schools and communities is targeting the right combination of actors with the greatest potential to influence students. In particular, training teachers to better understand and use the Life Skills Curriculum and supporting mothers’ groups to educate girls on sexual and reproductive health are seen as jointly promoting girls’ HIV/AIDS knowledge. Finally, Findings 8, 9, and 17 show that ASPIRE’s work addressing structural and cultural barriers to girls’ education through gender-sensitive WASH infrastructure and community engagement is seen as a highly relevant contribution. Stakeholders went beyond the sectoral silos, however, to articulate how successes under one output contribute to successes under other outputs (Finding 2). Simply put, ASPIRE’s value is greater than 1 ASPIRE’s cross-sectoral model is a strong program design, reflecting a holistic view of beneficiaries that resonates with all stakeholders. Although managing multiple funding streams is difficult in implementation for managers at all levels, the evaluation findings suggest that ASPIRE’s interventions have the potential to achieve its objectives and offer preliminary evidence supporting the development hypothesis. February 2018 | ASPIRE Performance Evaluation Report 80 the sum of the successes across outputs. For example, improvements in menstrual hygiene practices and infrastructure are seen as having improved girls’ attendance at school (Findings 2 and 17), while school health days and the school-based referral system for Youth-Friendly Health Services show that the school is an effective entry point for health service providers to reach adolescents, thereby supporting increased access to health services (Finding 6). In quantitative and qualitative data, students affirmed teachers and families as the top two influencers on their health knowledge and practices, affirming the importance of targeting both school and community settings (Finding 19). These results affirming the cross-sectoral design make intuitive sense, reflecting the way youth—and most adults, for that matter—view themselves and experience needs. The results also align with substantial research in the fields of education, health, and girls’ empowerment, which has repeatedly found that holistic approaches are necessary to achieve progress in each area. In education, for example, the literature has highlighted the role of the community and parents in children’s academic achievement, both in terms of school participation and home-based support for learning and attitudes toward education (see, for example, Ginsburg et al. 2014; Cao and Ramesh 2014; Friedlander and Goldenberg 2016). Health studies have closely linked a need for multipronged approaches to effective health outcomes (Hallfors et al. 2017; Joshi et al. 2015). Finally, a recent comprehensive literature review on what works in girls’ education by the Brookings Institution catalogued seven key approaches to addressing girls’ education, of which five are directly relevant to the Malawian context and are being addressed by ASPIRE: (1) making schools affordable, (2) addressing girls’ health, (3) making schools girl-friendly, (4) improving the quality of education, and (5) increasing community engagement (Sperling and Winthrop 2016).12 Although ASPIRE’s holistic design is strong, it is not without management challenges in implementation. Findings 21 and 22, in particular, detail how multiple funding streams and sectors expand the number of indicators and reporting burden on implementing partner staff. As USAID staff noted, it also creates the potential for conflicting or competing messages around priorities if offices are not seen to “speak with one voice.” The influx of DREAMS funding midway through ASPIRE compounded this challenge, leading to additional reporting requirements, an increased number of USAID-funded partners to integrate with (Finding 13), and the need for flexibility to shift focus in activity planning; the funding increase also included geographic expansion. Extending all interventions to Zomba to maintain the holistic approach required creative activity planning because the additional DREAMS funding did not come with additional basic education and PEPFAR Orphans and Vulnerable Children funds (the expansion did include additional WASH funds for Zomba). As a result, this expansion stretched program staff’s time. ASPIRE’s and USAID’s ability to respond accordingly and maintain fidelity of the cross-sectoral model in the face of these challenges is a notable achievement. 12 The other two are reducing the time and distance to school, and sustaining education during emergencies. February 2018 | ASPIRE Performance Evaluation Report 81 Similarly, Finding 20 illustrates that integration with other USAID partners has contributed to ASPIRE’s successes, but can be difficult to manage. Finding 12 documents that introducing a PEPFAR DREAMS coordinator has helped alleviate this challenge by promoting integration across DREAMS partners; more broadly, shared objectives and joint partners’ planning meetings have promoted integration across USAID partners in general. The challenges of multiple funding streams and sectors offer a plausible explanation for some of the weaknesses noted in ASPIRE’s management, which manifested in different ways at different levels. Project-level stakeholders, for example, pointed to being overwhelmed with technical work and weaknesses in ASPIRE’s monitoring (Findings 15 and 21). School- and district-level stakeholders, as well as community members benefiting from training, spoke of the need for greater follow-up to activities already underway (Finding 15). Finding 3 documents adequate reading fluency rates in Chichewa and English, but although students seem to have learned the mechanics of reading, they do not understand the text they read. Even though reading fluency is highly correlated with reading comprehension, it is not the same. This is underscored by students’ poor performance in reading comprehension tasks for both languages. Students need further support to build the higher order skills of reading with comprehension. Weak English oral language skills could partially explain the weakness in reading comprehension. As illustrated in the literacy model in Exhibit 46, spoken language is a foundational skill that correlates broadly with reading and writing skills, and comprehension in oral language is a precursor to reading comprehension. The English listening comprehension task, which does not involve written text, provides an indication of a child’s grasp of the language and comprehension ability, independent of written reading skills such as decoding. Pairwise correlations between English listening comprehension and English reading comprehension reveal a moderate overall relationship between the two tasks (correlation coefficient=0.46, p<0.0001). Scores on English listening comprehension statistically explained 21 percent of the variation in English reading comprehension scores. Combined, these analyses offer evidence that poor oral language skills in English and a lack of comprehension strategies in general are plausible explanations for some of the weakness in English reading comprehension, although these analyses cannot confirm a causal relationship. 2 ASPIRE has made strong progress toward implementation targets across all three areas of the results framework. However, the current pace of change in outputs and the strategic development objective is unclear due to baseline limitations, making it difficult to determine whether ASPIRE will meet these targets in its performance period. Meanwhile, students’ health knowledge and reading fluency have yet to translate into positive health-seeking behaviors and reading comprehension. February 2018 | ASPIRE Performance Evaluation Report 82 Because most oral language growth for primary students continues to come from non￾print sources, such as family, peers, teachers, class discussions, television, or radio (Biemiller 2003; Otto 2015; Neumann et al. 2009), more attention to the oral skills of speaking and listening as well as vocabulary are likely to improve students’ foundational language skills and help prepare for their eventual transition to reading with comprehension. Similarly, comprehension— both oral and reading—is a complex cognitive task (Kim et al. 2016); fostering this skill requires explicit instruction in comprehension strategies (Shanahan et al. 2010). As such, curricular and instructional revisions to emphasize comprehension strategies hold the potential to bolster both reading and listening comprehension, and ultimately, help students begin “reading to learn.” Exhibit 46: Component skills in reading comprehension and their structural relationships A parallel can be seen in ASPIRE’s health results, where strong knowledge and positive attitudes are seen in topics covered by the Life Skills Curriculum (Finding 5), but are more limited in healthcare￾seeking practices, such as rates of voluntary HIV testing and counseling, and condom use among sexually active students (Findings 6 and 7). Knowledge and attitudes have yet to translate into safe sexual practices that protect girls from unwanted pregnancy and all students from HIV. Knowledge of HIV transmission is also somewhat narrow; although students correctly identify sexual intercourse and contaminated blood as main transmission avenues, large proportions of respondents remain unaware of mother-to-child transmission (Finding 5). This could be due to the students’ age or to limitations in the curriculum. The Life Skills Curriculum focuses heavily on abstinence education and only select methods for HIV transmission (primarily through sexual intercourse and contaminated blood), neglecting a strong focus on important topics such as prevention of mother-to child transmission. This is an important point for future training given that Malawi has one of the February 2018 | ASPIRE Performance Evaluation Report 83 highest rates of child marriage, with one in two girls married by age of 18 (UNICEF 2016), as well as some of the highest rates of HIV incidence among adolescents, with young people accounting for 50 percent of new infections (Small and Weller 2013). Although ASPIRE indicators track only students’ knowledge of condoms and abstinence as prevention methods, and 2017 KAP data confirmed their level of knowledge in these two areas is strong, there were important gaps in their knowledge of other important prevention methods, including voluntary medical male circumcision and limiting the number of sexual partners. Again, this gap aligns the Life Skills Curriculum, which limits teaching on ways to prevent HIV. It has been well documented that there are multiple ways to reduce the risk of HIV transmission in addition to condom use, including voluntary medical male circumcision and limiting the number of sexual partners. Moreover, teaching abstinence education only has been proven to be an ineffective method of reducing HIV risk behavior (Lo et al. 2016; Santelli et al. 2017). Finding 7 regarding contraceptive use underscores the amount of progress that remains to be made in promoting safe sexual and reproductive health practices, and aligns with the most recent Demographic and Health Survey (2015–2016), which shows injectable birth control (15 percent) and condoms (14 percent) as the most well-known forms of contraception (Malawi DHS 2015 – 2016). However, the concern remains that, overall, the use of contraceptives among sexually active students is limited. As seen in Finding 7, the fact that only a quarter of sexually active upper primary students and slightly more than half of sexually active secondary students are using contraceptives should be noted for future programming, particularly because this is a pronounced problem throughout Malawi. Based on the most recent DHS (2015–2016), only 26 percent of sexually active adolescents are using contraception. The top reasons sexually active unmarried adolescents do not use condoms in Malawi include because they are not married (59 percent), they are having sex infrequently (17 percent), or they fear side effects or health concerns from using contraception (13 percent) (Malawi DHS 2015– 2016). Because the barriers to contraceptive use differ within and across countries, guidance from organizations working in adolescent reproductive health tends to be high-level. The World Health Organization (WHO) and USAID recommend interventions that target three key areas: (1) influencing community members to support access to contraceptives for adolescents; (2) improving health service delivery to adolescents (often through youth-friendly health service provision); and (3) providing accurate information and education about contraceptives, in particular curriculum-based sexuality education (Scholl n.d.; HIPs 2015; WHO 2011). Although community- and school-level respondents strongly emphasized the importance of infrastructure and activities that support sanitation and menstrual hygiene management (Findings 8, 17 and 19), ASPIRE has only been able to scratch the surface of the structural needs (Finding 25). This is seen in the fact that ASPIRE is on track to meet implementation targets for WASH infrastructure development (Exhibit 19 in Finding 2), but its own needs assessment revealed early on that resources would not be sufficient to address the needs (Finding 25). February 2018 | ASPIRE Performance Evaluation Report 84 Even though results indicate that ASPIRE has offered a strong response to the cultural barriers to girls’ education discussed under Output 3, challenges remain. Attitudes and practices seem to have improved in several areas. Nevertheless, some students noted that families and communities do not fully support the health and education needs of girls. However, social attitudes are complex in nature and tend to change slowly. Policy responses have helped shed light on the issues at the community level, but even successful sensitization combined with policy work cannot change attitudes quickly. As discussed under Finding 9, some local policies regarding local chiefs fining families whose daughters become pregnant have also proved to be worrisome, although well intentioned. These punitive measures could have unintended consequences for girls and families, and should be avoided. ASPIRE reported that these policies have been clarified to fine parents only if a girl becomes pregnant or gets married upon persuasion of the parents, but this was not reflected in performance evaluation data, perhaps owing to the evaluation’s timing. The main concern among most respondents was the sustainability of ASPIRE’s activities. Many ASPIRE’s activities focus on providing physical or monetary support for girls, such as education bursaries supported through DREAMS funding, sanitation facilities construction through WASH funding, and seed funding to mothers’ groups. Many respondents highlighted the concern that the successes made under ASPIRE might be threatened when this funding ends, while at the same time, they also expressed a desire for additional direct contributions of this kind (Finding 25). Finally, it is still early to draw conclusions regarding ASPIRE’s progress toward its development objective of decreasing girls’ dropout and repetition rates. Baseline limitations and weakness in administrative dropout and repetition data compound the difficulty of drawing conclusions on progress toward these goals. As captured in Finding 1, the limited quantitative data available show mixed results, although qualitative data indicated local stakeholders are seeing progress. Ultimately, girls’ likelihood to dropout or repeat grades results from a series of factors compounded over years of girls’ experiences, both inside and out of the education system; as such, decreasing these rates will be a slow process and even moderate success is notable in the space of 2.5 years. 6.2 RECOMMENDATIONS 6.2.1 RECOMMENDATIONS FOR ASPIRE  Continue implementing high-quality interventions that reflect the cross-sectoral strengths ASPIRE brings. Training, materials, sensitization campaigns, and work addressing structural barriers have all won praise, and the focus on quality and the holistic approach should be maintained. However, if training remains a focus, an increase on monitoring of training quality and outcomes should be included in ASPIRE’s activities.  Continue collaboration with government officials and integration with other USAID activities, including DREAMS partners. Collaboration with government won praise from activity stakeholders, as seen in Finding 14, while the success of integrating with other USAID activities was highly evident in interventions such as school health days and the February 2018 | ASPIRE Performance Evaluation Report 85 referral system. Further improving communication and coordination with district government bodies, such as the District Education Network, could help grow the dividends of the cross-sectoral model even more. Recognizing that school- and community-level participants are often involved in multiple projects, strategies that make communication clearer for these stakeholders could help smooth implementation processes at the local level.  Continue to support community-level actors who work across sectors. However, ensure that ASPIRE messaging is not unintentionally translated into negative outcomes. For example, ASPIRE should continue communicating that the punitive measures noted under Finding 9 are not the intended solutions to child marriage, to guarantee that communities’ solutions do not harm the beneficiaries they are intended to help. Work with traditional leaders and village chiefs to, instead, encourage positive healthcare-seeking behavior during initiation ceremonies, a key time in adolescent learning about their sexual and reproductive health, rather than threatening financial repercussions.  Strengthen indicator-monitoring systems that feed into reporting with a goal to improve data quality and report clarity; this will help ensure results are clear to stakeholders outside the immediate implementing team. Results should indicate timing of data collection, note whether values are cumulative, and provide sufficient detail on indicators so health and education specialists can easily interpret results. In particular, literacy results should make clear the competency and language assessed, as well as how scores are calculated. Making the existing evidence more accessible in this way can help build understanding of successes, and contribute to momentum and sustainability in ASPIRE’s final year. District government officials appear particularly eager for this information. Beyond improving data quality for indicator reporting, ASPIRE should ensure that monitoring systems are designed to assess the quality of activities on an ongoing basis and provide up-to-date feedback to stakeholders, particularly at school, community, and district levels. Quality monitoring is especially important for training programs, which would benefit from routine collection of participant feedback and post-training follow-up.  Introduce or modify activities that will capitalize on the existing strong knowledge and attitudes, and translate these into positive practices for both literacy and health. In education, for example, classroom and extracurricular instruction clearly needs practical models to help translate fluent reading into reading comprehension, including explicit instruction of diverse comprehension strategies. Explore how these strategies can target girls’ comprehension in particular to help close the gaps identified in Finding 3. In health, interventions should target helping students turn knowledge into practice. This might be challenging within schools, given the constraints of the existing Life Skills Curriculum. Consequently, it will be important to consider ways to disseminate these messages through community-based programming and ensure that policymakers understand the limitations of the current curriculum. February 2018 | ASPIRE Performance Evaluation Report 86  Since ASPIRE has provided a number of direct inputs in the first 2.5 years, outline and implement models for sustainability in the final year to ensure that district, school, and community structures can sustain interventions once direct financing and support end.  Use internal monitoring, evaluation, and learning activities to help answer new questions raised by this evaluation in ASPIRE’s final year. Two particular questions emerge as important to informing future work: (1) What factors are driving differences in girls’ and boys’ oral and reading comprehension? and (2) What interventions hold the greatest potential to translate HIV/AIDS and sexual and reproductive health knowledge into positive practices and healthcare-seeking behaviors? 6.2.2 RECOMMENDATIONS FOR USAID/MALAWI  Continue investing in and supporting implementation of a holistic approach to girls’ empowerment activities that reflect understanding of beneficiaries’ needs and context. The cross-sectoral activity design reflected in ASPIRE is a model for future activities in this area, and should be expanded in the next cycle of program planning. Events hosted by USAID/Malawi that explicitly foster integration across USAID-funded activities have supported ASPIRE’s integration with other activities in ways that have benefited results in multiple areas (and ultimately, beneficiaries); these should also be continued. USAID/Malawi offices should persist in seeking opportunities to support activities, drawing from multiple funding streams where these links are logical.  Support expansion of successful ASPIRE interventions to other districts where resources are available and prioritize where investments are made. Given the positive results attributed to WASH and DREAMS investments in school settings, USAID should look for ways to extend these areas of work while avoiding redundancies with other partners’ or donors’ work in the same geographic area. There are multiple schools where WASH infrastructure is a high priority, particularly menstrual hygiene management facilities, and Balaka could benefit from DREAMS-supported activities. Work with the Government of Malawi to determine its role in scale-up and align USAID programming with these efforts.  Strengthen internal management procedures and external communication for activities with multiple funding streams to ensure that offices agree on priorities and that USAID speaks with one voice in articulating these priorities. Activities with multiple funding streams could benefit from a champion at USAID who views them as a whole, rather than just those results supported by their office. Offices should ensure that they are mutually supportive in their communication to implementing partners, and view the project from the perspective of its ultimate intended goal, not just sectoral targets. Recognize the potential for new funding to complicate management for implementing partners, as was the case with DREAMS. Continue exercising an active coordination role to promote integration when a larger number of implementing partners are involved. February 2018 | ASPIRE Performance Evaluation Report 87  Approach activity monitoring and evaluation from a holistic perspective, considering the total reporting burden for the implementation team. Minimizing indicator revisions improves consistency of data, and enables better tracking and comparison of results over time. Approaching evaluation needs for the entire lifecycle at the outset of an activity can lead to more robust evaluations that draw deeper insights and are better able to find evidence of USAID’s contributions to the development objective. For example, limitations in internal baseline studies affect the strength of future evaluations; for future activities, USAID should ensure that baselines are of sufficient quality and documentation to allow effective comparison by future external evaluation teams. Planning should begin now for ASPIRE’s endline evaluation. Explore learning activities that can help answer the additional questions raised by this evaluation (listed in the recommendations to ASPIRE, above).  Support ASPIRE’s advocacy work, particularly around the Life Skills Curriculum, by reinforcing these messages in communication with the Government of Malawi. Ensuring that policymakers understand the constraints of the current curriculum will lead to more relevant interventions and more realistic expectations, and in the long run, could help produce openings for curricular revisions. 6.2.3 RECOMMENDATIONS FOR THE CENTRAL GOVERNMENT OF MALAWI  Use learning from ASPIRE to make appropriate decisions on policy, curricula, and scale-up. Continue supporting ASPIRE initiatives, particularly around operationalizing policy and curricula.  Scale up ASPIRE interventions that show promise in order to achieve national reach. This includes expanding use of the Life Skills Curriculum and considering use of the associated continuing professional development program ASPIRE developed to train teachers for more effective use of the MoEST’s curriculum. To promote adolescents’ access to health services, support and expand school health days and the school-based referral system; this may involve a combination of policy and communication work to operationalize guidance and support uptake. School WASH committees are another promising initiative; working with these committees can help achieve the government scale-up of the process of improving WASH infrastructure in schools. Similarly, to build on ASPIRE’s work to improve literacy instruction in upper primary school, consider expanding new programming components, especially continuing professional development for teachers.  In conjunction with scale-up, work with ASPIRE to create a sustainability plan during the last year of implementation aimed at ensuring institutional arrangements for uptake. Identify the ministry responsible for supporting ASPIRE’s community engagement work to ensure that work continues with WASH committees, traditional leaders, and particularly mothers’ groups. February 2018 | ASPIRE Performance Evaluation Report 88  Encourage cross-sectoral engagement at the national level, particularly in aligning policies across departments. For example, consider revision of the National Education Policy, which limits sexual and reproductive health education on school grounds (including not allowing HIV testing and contraceptive distribution in schools).  Convene stakeholders from all the relevant ministries and departments, along with subject matter experts, to comprehensively review and update the Life Skills Curriculum. This evaluation offers insights into strengths and limitations of the current curriculum that can serve as a starting point to identify potential revisions, and also provides evidence to support advocacy regarding the importance of these revisions.  Under the leadership of the MoEST, review and update the upper primary reading curriculum to align it with the National Reading Program used in lower primary. This process could involve updating literacy instruction in Standards 5–8 to build on and extend the curriculum and instruction used in Standards 1–4; these revisions should include emphasis on multiple comprehension strategies to help students learn to understand both written and oral language. Develop corresponding benchmarks for Chichewa and English reading skills; this evaluation provides some data that can be used in initiating benchmark conversations and identifying additional evidence needs for the benchmarking and curricular revision process. 6.2.4 RECOMMENDATIONS FOR DISTRICT-LEVEL GOVERNMENT OFFICES  Continue to host coordination meetings at District Executive Committee to encourage collaboration among implementing partners.  During the last year of implementation, work with ASPIRE to create a sustainability plan aimed at taking over some of the successful activities ASPIRE has been implementing.  Provide feedback to central Government of Malawi ministries that are supporting promulgation of successful ASPIRE activities in order to ensure that policies, guidance, and support are relevant and user-friendly. February 2018 | ASPIRE Performance Evaluation Report 89 REFERENCES Biemiller, A. 2003. Vocabulary: Needed if more children are to read well. Reading Psychology 24: 323–335. Bisika, T., P. Ntata, and S. Konyani. 2009. Gender-violence and education in Malawi: a study of violence against girls as an obstruction to universal primary school education. Journal of Gender Studies 18(3): 287–294. Available at: http://dx.doi.org/10.1080/09589230903057183 Cao, Y. and A. Ramesh. 2014. Parent and Community Support for Early Grade Reading. Webinar. Global Reading Network (December). Available at: https://globalreadingnetwork.net/resources/parent-and-community-support-early-grade-reading Friedlander, E., and C. Goldenberg, eds. 2016. Literacy Boost in Rwanda: Impact Evaluation of a Two Year Randomized Control Trial. Stanford, CA: Stanford University. Available at: https://static1.squarespace.com/static/57ffc29c414fb543385340da/t/580b907f6b8f5b0d54ca46 4a/1477152950891/Friedlander_Goldenberg_2016_LiteracyBoostInRwanda.pdf Ginsburg, M., R. Klauss, F. Nankhuni, L. Nyirongo, L. J. Sullivan Omowoyela, E. Richardson, R. Terwindt, and C. Willimann. 2014. Engaging community members in enhancing educational quality: Studies of the implementation of the Primary School Improvement Programme in Malawi. Southern African Review of Education 20(1): 30–57. Hallfors, D., H. Cho, S. Rusakaniko, B. Iritani, J. Mapfumo, and C. Halpern. 2011. Supporting adolescent orphan girls to stay in school as HIV risk prevention: evidence from a randomized controlled trial in Zimbabwe. American Journal of Public Health 101(6) (2011): 1082–1088. High-Impact Practices in Family Planning (HIPs). 2015. Adolescent-friendly contraceptive services: mainstreaming adolescent-friendly elements into existing contraceptive services. Washington, D.C.: USAID. Available at: https://www.fphighimpactpractices.org/briefs/ adolescent-friendly￾contraceptive-services/ Joshi, J., G. Buit, and D. González-Botero. 2015. Menstrual Hygiene Management: Education and Empowerment for Girls? Waterlines 34(1): 51–67. Available at: http://www.developmentbookshelf.com/doi/pdf/10.3362/1756-3488.2015.006 Kim, Y.-S. G., H. N Boyle, S. S. Zuilkowski, and P. Nakamura. 2016. Landscape Report on Early Grade Literacy. Washington, D.C.: United States Agency for International Development. Available at: https://globalreadingnetwork.net/sites/default/files/research_files/LandscapeReport.pdf Lo, N. C., A. Lowe, and E. Bendavid. 2016. Abstinence funding was not associated with reductions in HIV risk behavior in Sub-Saharan Africa. Health Affairs 35(5): 856–863. February 2018 | ASPIRE Performance Evaluation Report 90 Malawi Ministry of Health and Columbia University. 2016. Malawi Population-Based HIV Impact Assessment: MPHIA 2015–2016. Summary Sheet: Preliminary Findings. Available at: http://phia.icap.columbia.edu/wp-content/uploads/2016/09/MALAWI-Factsheet.FIN_.pdf Mellish M., S. Settergren, and H. Sapuwa. 2015. Gender-Based Violence in Malawi: A Literature Review to Inform the National Response. Washington, D.C.: Futures Group, Health Policy Project. Available at: https://www.healthpolicyproject.com/pubs/436_FINALHPPMalawiGBVLiteratureReview.pdf National Statistical Office (NSO) [Malawi] and ICF. 2017. Malawi Demographic and Health Survey 2015-16. Zomba, Malawi, and Rockville, Maryland, USA: NSO and ICF. Available at: https://dhsprogram.com/pubs/pdf/FR319/FR319.pdf National Statistical Office (NSO) and ICF Macro. 2011. Malawi Demographic and Health Survey 2010. Zomba, Malawi, and Calverton, Maryland, USA: NSO and ICF Macro. Available at: https://dhsprogram.com/pubs/pdf/FR247/FR247.pdf Neumann, M. M., M. Hood, and D. L. Neumann. 2009. The Scaffolding of Emergent Literacy Skills in the Home Environment: A Case Study. Early Childhood Education Journal 36(4): 313–319. Otto, B. 2015. Literacy Development in Early Childhood: Reflective Teaching for Birth to Age Eight. Longrove, IL: Waveland Press, Inc. Pendleton, J., M. Mellish, and H. Sapuwa. 2016. PEPFAR Malawi Gender Assessment Report (2015). Washington, D.C.: Palladium, Health Policy Plus. Available at: http://www.healthpolicyplus.com/ns/pubs/2006- 2007_PEPFARMalawiGenderAssessmentFINAL.pdf. Scholl, Ed. n.d. Youth and Contraception: Needs and Challenges. Last revised: 12/09/04. Global Technical Briefs. Available at: https://www.k4health.org/sites/default/files/youth%26contracep_eng.pdf Shanahan, T., K. Callison, C. Carriere, N. K. Duke, P.D. Pearson, C. Schatschneider, and J. Torgesen. 2010. Improving reading comprehension in kindergarten through 3rd grade: A practice guide (NCEE 2010-4038). Washington, D.C.: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Available at: https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/readingcomp_pg_092810.pdf Small, E. and B. Weller. 2013. Profiles of Malawian adolescents at risk for HIV infections: Implications for targeted prevention policy, and practices. AIDS Care 25(3):289-295 Sperling, G., and R. Winthrop. 2016. What Works in Girls’ Education: Evidence for the World’s Best Investment. Washington, D.C.: Brookings Institution Press. United Nations Children’s Fund (UNICEF). 2016. The State of the World’s Children 2016: A fair chance for every child. Available at: https://www.unicef.org/sowc2016/ February 2018 | ASPIRE Performance Evaluation Report 91 UNICEF. The State of the World’s Children 2016. Available at: https://www.unicef.org/sowc2016/ United States Agency for International Development (USAID) Malawi. Country Development Cooperation Strategy 2013–2019 (Approved March 25, 2013 through March 25, 2018, extended on May 4, 2017 through September 30, 2019). Available at: https://www.usaid.gov/malawi/cdcs World Health Organization (WHO). n.d. Malawi: Analytical summary (beta version). Regional Office for Africa, African Health Observatory. Available at: http://www.aho.afro.who.int/profiles_information/index.php/Malawi:Analytical_summary_- _Health_Status_and_Trends World Health Organization (WHO). 2011. Guidelines on Preventing Early Pregnancy and Poor Reproductive Outcomes Among Adolescents in Developing Countries. Available at: http://www.who.int/immunization/hpv/target/preventing_early_pregnancy_and_poor_reproductive _outcomes_who_2006.pdf February 2018 | ASPIRE Performance Evaluation Report 92 ANNEX 1: EVALUATION SCOPE OF WORK DESCRIPTION/SPECIFICATION/STATEMENT OF WORK PERFORMANCE EVALUATION OF GIRLS EMPOWERMENT THROUGH EDUCATION AND HEALTH (ASPIRE) PROJECT IN MALAWI 1. INTRODUCTION AND BACKGROUND Performance evaluation of the following project: 1 Project Title: USAID/Malawi Empowering Girls through Education and Health Activity (ASPIRE) 2 Award Number: AID-612-A-15-00001 3 Implementing Organizations: Save the Children International; Malawi Institute of Education; FAWEMA and Creative Centre for Community Mobilization (CRECCOM). 4 Award Dates: December 17, 2014 and ends on December 16, 2018. 5 Funding: $18.6 Million 6 Geographic Coverage: Zomba, Balaka and Machinga Districts, covering all primary and secondary schools 7 Targeting students and teachers: from Standards 4-8 and Forms 1-4. 8 Implementation Partners: Malawi Institute of Education (MIE), a parastatal organization; the Ministry of Gender, Children and Social Welfare (MGCSW); the Ministry of Health, the Ministry of Water development; and directorates from the Ministry of Education Science and Technology (MoEST) such as: Department of Inspection and Advisory Services (DIAS), the Department of Teacher Education and Development (DTED), the Department of Basic Education (DBE), the Department of Secondary Education, the Department of School Health and Nutrition (SHEN) and the Department of Education Planning (DEP). The ASPIRE project improves achievement of girls in upper primary and secondary schools. This is done through implementation of activities aimed at achieving results described below: Reading skills for girls in upper primary school improved: ASPIRE provides Standard 4-8 teachers with methodologies that improve reading instruction. The reading instruction improvements are done through in￾service trainings that train teachers on teaching reading effectively leveraging foundational literacy skills from lower and junior primary school level. In addition, ASPIRE provides technical assistance to MoEST in production and adaptation of grade level appropriate story books and story cards to ensure that students have adequate reading books that supplement the National Primary Curriculum textbooks to enable reading variety and that students have an opportunity to practice reading through a variety of reading titles. Further to this, ASPIRE helps schools in Balaka and Machinga with instructional leadership through continuous coaching and mentoring of teachers on appropriate and updated reading instruction techniques that teachers use to improve the reading performance of their students in English and Chichewa. Adoption of positive sexual and healthcare seeking behaviors among youth10-19 increased: ASPIRE empowers girls between the ages of 10-19 to adopt positive and health-care seeking behaviors through life skills and sexual and reproductive health education to influence reduction of early pregnancy and Human February 2018 | ASPIRE Performance Evaluation Report 93 Immunodeficiency Virus (HIV) infections among adolescents so that girls can remain in school. This involves increasing the number of beneficiaries who are Orphans and Vulnerable Children (OVC), PEPFAR OVC programs for children and families affected by HIV/AIDS, the provision of Sexual and Reproductive Health (SRH) education, improving prevention of sexual transmission of HIV/AIDS, and access to HIV/AIDS services. Key structural and cultural barriers for girls aged 10-19 decreased: This aims at working with the communities to address social norms for adolescents and decreasing key structural and cultural barriers for girls aged 10-19 access to schooling. It includes mobilizing communities towards keeping girls in school through knowledge empowerment in adolescent sexual and reproductive health rights and economic empowerment through village savings and loan groups. It also includes provision of water and sanitation (WASH) and menstrual hygiene facilities in schools. Cross-Cutting Issues USAID Malawi has three crosscutting issues: 1) Capacity of institutions improved, 2) Use of technology and innovation increased, 3) Policy and systems strengthened. ASPIRE addresses three cross-cutting issues outlined in the USAID/Malawi Country Development Cooperation Strategy (CDCS) by working through the Government of Malawi (GoM) structures, local organizations and communities to institutionalize best practices that help improve reading skills, increase access to sexual and reproductive health for adolescent girls, and empower communities to adopt norms that support and keep girls in schools as a way of developing resilience and sustaining the activity implementation. Furthermore, the activity innovatively engages with the private sector nurturing a mutual interest around getting more girls educated as a population and/or agents that will grow private sector business interests in the near future. ASPIRE Theory of Change The ASPIRE hypothesis postulates that if it supports the improvement of teaching and learning with quality reading instruction, provision of teaching and learning materials, adoption of positive sexual and healthcare seeking behaviors among youth10-19, parent and community engagement in addressing key structural and cultural barriers for girls aged 10 -19 then achievement of learners, especially girls, in upper primary and secondary schools will improve. This will result in overall gains in student performance, lower drop-out and repetition rates, and increased persistence through the eight standards of primary school and four years of secondary school for both boys and girls. ASPIRE Results Framework The ASPIRE results framework is presented in the figure on the next page that highlights the pathways through which the ASPIRE implementation attains the ASPIRE development hypothesis, goals, outcomes and outputs. February 2018 | ASPIRE Performance Evaluation Report 94 February 2018 | ASPIRE Performance Evaluation Report 95 2. SCOPE/PURPOSE OF THE PERMANCE EVALUATION The Contractor must establish ASPIRE’s progress against its objectives, propose adaptation and remedial measures for ASPIRE’s implementation, and produce lessons learned for application in future programming. The focus of the evaluation will be analyzing the evaluation questions at the level of the individual activity components that comprise ASPIRE including the crosscutting issues. This will inform future implementation of the activity and similar designs in future. Specifically, the Contractor must do the following:  Identify implementation challenges, unmet needs, and unintended consequences or impacts of ASPIRE.  Provide a better understanding of the progress made by each component of the ASPIRE on issues such as relevance, sustainability and cost-effectiveness.  Confirm if the performance of ASPIRE will attain the underlying development hypotheses; and  Provide recommendations for any course corrections necessary to achieve the ASPIRE’s hypothesis. 3. EVALUATION QUESTIONS The ASPIRE performance evaluation will answer the following questions: i. To what extent is the ASPIRE design and its implementation on course to achieve the ASPIRE development objective? ii. How is ASPIRE integrating with other USAID funded activities and other development partners in Balaka and Machinga? iii. How is the ASPIRE activity coordinating with district government bodies? What gains is ASPIRE achieving through its engagement with the private sector? What are the most significant accomplishments, best practices, and lessons learned from the ASPIRE activity? vi. How does ASPIRE need to adapt its approach in order to achieve its objectives? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above evaluation questions. 4. TASKS The Contractor must perform the following tasks:  Develop evaluation model,  Test and verify the evaluation model,  Deploy a field team,  Collect the relevant data to inform the evaluation,  Present conclusions and recommendations for each study question to USAID/Malawi prior to drafting the final report; and  Conduct workshop to present the draft evaluation findings to relevant stakeholders for validation of findings. 5. DELIVERABLES AND TIMELINE The Contractor must provide the following deliverables with the requirements highlighted below: A) Inception Report February 2018 | ASPIRE Performance Evaluation Report 96 The inception report must describe the conceptual framework the Contractor will use to undertake the evaluation. It will detail the evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources, sampling and indicators) and address all technical requirements. The Inception report should not exceed 15 pages, excluding annexes and at a minimum must contain the following:  Evaluation Methodology: The evaluation design must include appropriate sample sizes required to ensure minimum detectable size effect to allow for required scientific rigor and describe and document the methodological approaches that will be used to answer the evaluation questions sufficiently and clearly. The design must include an evaluation framework and assessment tools for each evaluation question and highlight the conceptual model(s); and specify the measurement criteria to be used to respond to each question. It must discuss any risks and limitations that may undermine the reliability and validity of the evaluation results. The design must outline data collection processes for each question.  Complete set of evaluation questions, elaborated on as necessary.  Discussion of risks and limitations that may undermine the reliability and validity of the evaluation results.  Specification of indicators that must be used as a guide in answering each question.  Discussion of the data collection and data analysis methods that will be used for each question. State the limitations for each method. Include the level of precision required for quantitative and qualitative methods and value scales or coding used for qualitative methods. Standard data collection methods for USAID evaluations are: surveys, questionnaires, interviews, focus groups, document review and observations.  Detail key data sources that will be selected to answer each evaluation question.  Explanation of how existing data will be incorporated and used to answer the evaluation questions.  Timeline showing the evaluation phases (data collection, data analysis and reporting) with their key deliverables and milestones.  Specific responsibilities of each team member for each evaluation phase. Include any changes in the evaluation team.  Discussion of logistics for carrying out the evaluation. Include specific assistance that will be required from USAID, such as providing arrangements for key contacts within the Mission or Government.  Discussion on the use of spatial data collection methods and formats to ensure locations included in the evaluation sampling frame are captured for integration into the Mission’s geographic information system and to permit spatial analysis of evaluation data at the school level. The Contractor must provide georeferenced data sets to the USAID/Malawi TOCOR. The data must be provided in an MS Excel sheet that includes a unique identifier for each data record, with latitude and longitude locations in decimal degree format to the fifth place (e.g., 34.45673 and -13.36712). During the inception plan the COR will work closely with the Contractor to determine other applicable evaluation data that will be included into the spatial data table for each evaluation location. USAID/Malawi will provide a generic EXCEL template for the Contractor to use that will facilitate this process.  Appended draft instruments for data collection specific to questions and indicators in the evaluation.  The inception report must clearly document and discuss how gender and disability analysis will be integrated into the design of the evaluation. B) Draft Evaluation Report The Contractor must submit an evidence-based Draft Evaluation Report that gives a clear and concise answer to each evaluation question with relevant explanation and supporting information. The Mission will have 10 working days to review the draft report. The Contractor will have 10 working days after receipt of the Mission’s comments to address the comments to the satisfaction of the Mission. The TOCOR will communicate the approval to the Contractor when it is attained. C) Findings Workshop February 2018 | ASPIRE Performance Evaluation Report 97 The Contractor must use either a cover memorandum or similar format to explain how comments provided by the Mission were addressed and how it differs substantially from the initial version. After incorporating USAID’s comments into the draft Evaluation Report, the Contractor must present the key findings, conclusions and recommendations at a half-day workshop no more than five weeks after the Draft Evaluation Report is approved by USAID/Malawi. The workshop must be held in Lilongwe and is anticipated to be attended by between 25 to 35 key stakeholders. The Contractor must be responsible for costs, logistics and managing invitations to this workshop. At least 10 of the representatives must be from sampled districts outside of Lilongwe. The Contractor must produce a summary/briefer (maximum five pages) of key findings, conclusions and recommendations to be distributed to stakeholders during the workshop. D) Final Evaluation Report The Contractor must submit an evidence-based Final Evaluation Report that answers, in full, each evaluation question and incorporates any relevant information resulting from discussions from the findings workshop. The report must comply with the Checklist for Assessing USAID Evaluation Reports and the technical requirements that will be provided to the Contractor by the TOCOR. The final report format will be agreed with the USAID/Malawi Education Office, but should at minimum include:  USAID branded cover page,  Executive summary (3-5 pages summarizing key points, including project purpose and background, key evaluation questions, methods, findings, conclusions, and recommendations),  Data Methods and Analysis,  Findings, Conclusions and Recommendations; and  Appendices as appropriate. USAID/Malawi/TOCOR will approve the final report. The report must be in English, should not exceed 50 pages excluding relevant Annexes, (e.g. SOW, interview transcripts/notes, photos and success stories), and must include matrices and other visuals to consolidate and summarize data. In addition, the Contractor must submit electronically to the TOCOR all tools, data sets and final evaluation report for USAID records. The Contractor must ensure that Appendix One of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. This includes:  The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the activity, what did not and why;  Evaluation reports must address all evaluation questions included in the scope of work;  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer;  Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report;  Evaluation findings will assess outcomes and impact on men, women, girls, and boys as well as people with disabilities;  Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.);  Evaluation findings must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence;  Sources of information must be properly identified and listed in an annex;  Recommendations must be supported by a specific set of findings; and  Recommendations must be action-oriented, practical and specific, with defined responsibility for the action. February 2018 | ASPIRE Performance Evaluation Report 98 6. EVALUATION TEAM In order to mitigate the perception or reality of biased measurement or reporting due to conflict of interest the Contractor team must comprise personnel external to management or implementation of the ASPIRE activity. The Contractor team should have a strong background in research, monitoring and evaluation. The Contractor must have an experienced Team Leader to lead the evaluation team and liaise with USAID and other stakeholders in conducting the evaluation. Gender must be considered in the formation of an evaluation team. The contractor must consider also using local personnel’s expertise. The contractor must consider proposing a staffing configuration that has a junior-level administrative assistant to facilitate logistics for the Evaluation Team (e.g., helping to contact potential interviewees and set up interview appointments, especially if a potential interviewee needs to be interviewed at a distant location). While the entire team will be responsible for all in-country logistical support, the Overall Team Leader will have the primary responsibility as the Point of Contact between the team and the USAID mission. 7. INFORMATIONAL RESOURCES PROVIDED BY USAID/MALAWI The following documents will be provided by USAID/Malawi to the Contractor for use during the performance evaluation:  Empowering Girls through Education and Health (ASPIRE) work plans (Year 1 and Year 2),  ASPIRE baseline Study Report,  Malawi Basic Education Statistics (Malawi Education Information Management System),  Maps of ASPIRE Impact Areas,  USAID/Malawi CDCS 2013-2018,  ASPIRE Program Description,  ASPIRE Activity Monitoring and Evaluation Plan,  Quarterly and Annual Reports; and  As warranted, the Contractor will receive additional project-related documentation. February 2018 | ASPIRE Performance Evaluation Report 99 ANNEX 2: DETAILED DESIGN, METHODS, AND LIMITATIONS This external performance evaluation, designed as an accountability and learning tool for USAID/Malawi and ASPIRE activity stakeholders, integrated quantitative and qualitative methodologies within a utilization-focused approach that engaged key stakeholders in evaluation planning and results validation. EnCompass LLC proposed an initial evaluation design in its technical response to the Request for Task Order Proposal. The evaluation team revised this proposed design based on stakeholder input obtained during inception and presented the final evaluation design in the Inception Report (June 19, 2017). The most significant revision was the decision to not use baseline data due to the baseline limitations noted in Section 2.5, below. After submission of the draft performance evaluation report in September 2017, this decision was revisited at the request of USAID/Malawi. This ASPIRE 2017 Performance Evaluation Report presents baseline information where data exist, but without conducting statistical comparison of 2017 performance evaluation data and 2015 baseline data. As such, this evaluation refrains from using baseline data to draw findings and conclusions, notes limitations where the baseline data are presented, and visually offsets baseline analysis by presenting this information in grey text boxes. The final design presented in the Inception Report specified a sequential integration of quantitative and qualitative methods (a sequential, mixed-methods design), whereby quantitative data would be collected in Round 1, followed by a qualitative Round 2 after initial analysis provided emergent themes from the quantitative data. However, a strike by the Teachers’ Union of Malawi, from June 5 to 19, 2017, delayed Round 1 data collection. To complete data collection during the 2016–2017 school year and maintain the evaluation report schedule, the sequential design was abandoned, and qualitative and quantitative data collection overlapped. The information presented in the remainder of this section describes the final achieved design that resulted from these adaptations. 2.1 EVALUATION DESIGN This performance evaluation used a mixed-methods design that provides both breadth (via quantitative data) and depth (via qualitative data) to answer the evaluation questions. This design offers the flexibility not only to capture what achievements are occurring at a generalizable level, but also to gain a deeper understanding of why change is or is not happening and how achievements were made. Primary-source quantitative and qualitative data were supported with monitoring data and implementation information obtained through a document review. Together, these data types better informed results and led to more nuanced recommendations by balancing a practical assessment of progress toward ASPIRE indicators with considerations of the complex social February 2018 | ASPIRE Performance Evaluation Report 100 dynamics the development hypothesis attempts to influence and the sustainability of the interventions. Quantitative data were collected through a school-based survey in June and July 2017, designed to produce generalizable indicator estimates that can be inferred to the population of the intervention area. These primary source quantitative data reflect a single cross-section—ASPIRE activity beneficiaries at the time of the evaluation. As a short performance evaluation occurring midway in the activity’s implementation, a design featuring a counterfactual was not possible. Additionally, statistical estimation of change over time was not possible due to baseline limitations. The limitations of the single cross-sectional design reflected by the quantitative data are discussed in Section 2.5, below. EnCompass collected qualitative data from a purposive sample targeting schools and their surrounding communities. This design enabled triangulation of results at the local level during data collection, reflecting the ASPIRE’s development hypothesis and theory of change, which hold that sociocultural structures outside the schools are key enablers and barriers to girls’ empowerment. The purposeful sample was designed to capture the breadth of beneficiary and stakeholder perspectives, recognizing that idiosyncratic contextual factors could have strong differentiating effect on beneficiaries’ experiences and ASPIRE’s results; Section 2.3.2 provides detail on purposive sampling categories. 2.2 DATA COLLECTION TOOLS AND METHODS This section describes the data collection methods and associated tools used for this evaluation. See Annex 8 for the complete set of tools. 2.2.1 DOCUMENT REVIEW The evaluation team developed (in Microsoft Word) and used a document review matrix, and accompanying guidelines and data extraction forms (in Microsoft Excel) to track documents collected and code information using a structured approach. The document review process extracted activity monitoring data and provided information on background, context, and ASPIRE activities, which the evaluation team triangulated with other data sources. The document review began prior to inception and continued through the conclusion of primary data analysis as new documents and data became available. 2.2.2 QUANTITATIVE TOOLS AND METHODS This performance evaluation used four quantitative data collection tools:  Reading Assessment: To obtain data on literacy skills of learners in Standards 5 and 6. This tool was administered orally and individually (to one student at a time) by a trained assessor who marked responses. Reflective of the ASPIRE interventions, the assessment contained six February 2018 | ASPIRE Performance Evaluation Report 101 tasks measuring literacy skills in Chichewa and English: Chichewa reading fluency, Chichewa reading comprehension, English reading fluency, English reading comprehension, English listening comprehension, and English dictation. For administration and scoring detail, see below.  KAP Survey Questionnaire: To assess sexual and healthcare-seeking behaviors among learners in Standards 5 and 6 (upper primary) and Forms 1 and 2 (secondary), as well as the drivers of behavior. The KAP instrument was a closed-ended (structured) survey questionnaire, administered individually by a trained assessor in private. In upper primary schools, the KAP was administered to the same learners sampled for the reading assessment. The questionnaire also collected background demographic information about the learners.  Head Teacher Questionnaire: To collect school-level information on ASPIRE activities and results across all three outputs. The head teacher questionnaire was a close-ended (structured) survey questionnaire administered individually by a trained assessor in private.  School-Based Checklist: To document improvements in school materials and infrastructure, including WASH facilities and menstrual hygiene management facilities. The checklist was based on structured observation, featuring a series of closed-ended items that a trained assessor observed during a tour of the school grounds. The observation was completed after the head teacher questionnaire with the head teacher or their designee. The evaluation team adapted the quantitative tools from ASPIRE’s baseline assessment, examining baseline tools for validity, reliability, and relevance to the performance evaluation questions at the item level. In conjunction with USAID, the team added, dropped, or modified questionnaire and assessment items based on these needs. For the reading assessment, this review included the addition of new tasks, removal of tasks that could not be validated on the evaluation timeframe, and writing assessor instructions, which were not documented in the baseline tools. Scoring of the Reading Assessment: The Chichewa and English oral reading tasks measured students’ ability to read a narrative text, assessing both the rate (i.e., speed) and accuracy of reading. In combination, rate and accuracy indicate a child’s reading fluency; the rate also indicates the level of automaticity in the cognitive processes of oral reading, which is an important influencer of comprehension. The tasks were timed, with children given 1 minute to read as much as they could. The Chichewa oral reading passage was 61 words and the English was 140 words. The Chichewa and English reading comprehension tasks were based on the oral reading passage: students were asked only those comprehension questions that pertained to the text they had read. After the student read the text aloud, the assessor asked up to five comprehension questions about the Chichewa passage and up to four comprehension questions about the English passage. Students’ results can be presented in two ways: an accuracy score is the percentage of questions the student answered correctly out of the number of questions they were asked; an absolute score is the percentage of questions the student answered correctly out of the total possible. Except where baseline data are presented, this report presents accuracy scores because this is a better measure of the extent to which students understand what they read. February 2018 | ASPIRE Performance Evaluation Report 102 A text similar to the English reading comprehension text was used to assess English listening comprehension. The assessor read the text aloud and then asked the student five comprehension questions about the passage. The score is expressed as a percentage of the total correct out of five. Finally, in the English dictation task, students were first read a seven-sentence story. They were then read the story again, slowly, one sentence at a time, and asked to write down the words in each sentence. In total, the story included 71 words that students were expected to write in this task; the total score represents the percent of words written correctly, out of 71. 2.2.3 QUALITATIVE METHODS The performance evaluation used two qualitative methods:  Key Informant Interviews: To enable deep exploration of information responding to evaluation questions from the perspective of a key respondent. Interviews followed a semi￾structured guide that enabled probing of topics of interest, unique perspectives, and exploration of issues of particular importance to the respondent. This in-depth method built “strong” narratives that provide a holistic understanding of stakeholders’ experiences with ASPIRE in a private setting, allowing for candid exchanges.  Focus Group Discussions: To spur discussion in a group setting that enables building collective narratives around the evaluation questions and to gain insight into differences in perceptions among beneficiaries. Group discussions followed a semi-structured guide that encouraged conversation among participants and probing of topics that arose from these conversations. Interviews and discussions took place in a mix of Chichewa and English, depending on participants’ preferences and comfort with each language. Both methods solicited in-depth information about enablers and constraints to girls’ empowerment at each level of the ASPIRE’s theory of change; perceptions regarding ASPIRE’s relevancy and the factors influencing results, successes and challenges faced by ASPIRE in its first 2.5 years; and recommendations for ASPIRE’s final year. The interview and discussion guides collected information across all three outputs in the ASPIRE’s results framework and the overall development objective. Tools drew on appreciative evaluation techniques to elicit and learn from successes. Interview and discussion group guides were semi-structured to enable probing as topics arose and tailored to each respondent group to account for the respondents’ unique position within ASPIRE’s activities. Section 2.3.2 provides respondent categories. The evaluation team designed the interview and discussion guides specifically for the performance evaluation to take into account the timing of data collection (occurring close to the midpoint of the ASPIRE activity) and focus on the evaluation questions. Qualitative experts from EnCompass and evaluation team members in Malawi reviewed tools to ensure appropriateness to the context and adherence to best practice in qualitative methods. USAID reviewed draft and final tools to ensure February 2018 | ASPIRE Performance Evaluation Report 103 relevancy to evaluation needs, and evaluation team members piloted and revised the tools prior to data collection. 2.2.4 DATA COLLECTION PROCEDURES, TRAINING, AND ETHICS Desk review was conducted under the supervision of an evaluation specialist in EnCompass’ home office. Quantitative data were collected by the University of Malawi Center for Social Research, a subcontractor to EnCompass, under the leadership and supervision of EnCompass evaluation team members. Qualitative interviews and group discussions were facilitated by EnCompass evaluation team members experienced in qualitative methods, assisted by senior research assistants from the Center for Social Research, serving as dedicated notetakers. All data collection teams included a team supervisor who ensured adherence to sampling and tool administration procedures, including ethical procedures. On qualitative teams, the EnCompass team member served as the supervisor. All data collectors participated in a 5-day training workshop on quantitative tools and a 3-day training workshop on qualitative tools. Training components included data collection quality control, data management, and evaluation ethics, with extensive opportunities for practice in tool administration, including a real-world pilot-practice day in the middle of the workshop. Participants’ feedback from this pilot was used to ensure validity of the instruments before data collection began. The reading assessment training included a simulation of tool administration under controlled conditions to promote inter-rater reliability. To minimize processing error and improve data reliability, all quantitative tools were administered on tablets, enabling extensive checking/routing restrictions to minimize interviewer error (response error) and automating data entry; the reading assessment was programmed in Tangerine® and the other three quantitative tools were programmed in the Census and Survey Processing System (CSPro). The team lead, with the support of EnCompass technical specialists, monitored (rechecked) data quality in real time during quantitative data collection, communicating any patterns of administration errors to the data collection team supervisor so they could be addressed directly with the data collector. To ensure qualitative data accuracy, audio recordings were taken when respondents consented. The facilitator and notetaker transcribed, translated (as appropriate), and validated notes prior to data analysis, using audio recordings to clean and verify the accuracy of transcripts where available. The evaluation protocol underwent an Institutional Review Board appraisal by the Malawi National Commission for Science and Technology, which issued ethics approval for the evaluation before data collection began. All primary source data collection tools included an informed consent statement or, in the case of minors, an assent statement. Quantitative tools and all data collection with minors did not record respondents’ names or any other potential personally identifiable information. In quantitative datasets, variables that could be identifying information in conjunction with other variables (e.g., school name) were masked (anonymized) before datasets were transferred to USAID. Qualitative datasets were available only to the EnCompass evaluation team. All raw data were permanently transferred to secure EnCompass servers prior to completion of the evaluation and all February 2018 | ASPIRE Performance Evaluation Report 104 other copies, physical and electronic, were destroyed. All audio recordings were destroyed after transcripts were finalized, with no copies retained by any data collector or EnCompass. 2.3 SAMPLING DESIGN 2.3.1 QUANTITATIVE SAMPLE DESIGN AND WEIGHTING Quantitative data collection relied on a two-stage stratified random sampling procedure to select schools and students, with schools serving as clusters. Because ASPIRE interventions target the population of schools in the intervention districts, the sample drew from all registered schools in that district; there is no separation between intervention and comparison schools. Sampling stages using the following procedures:  Stage 1: A random sample of schools was drawn using probability proportional to size, stratified by district. MoEST administers Zomba as two separate districts (urban and rural), leading to a total of four strata at the first stage. The MoEST EMIS school lists provided by the districts constituted the Stage 1 sampling frame.  Stage 2: At each school selected in the first stage, a simple random sample was drawn of learners in the targeted grades, stratified by sex and grade. In primary schools, the target grades were Standards 5 and 6; in secondary schools, the target grades were Forms 1 and 2. The students present on the day of data collection constituted the sampling frame. To ensure an unbiased sample, attendance was compared with official school enrollment figures prior to Stage 2 sampling. Because ASPIRE interventions and expected results differ between primary and secondary schools, separate samples were drawn for each school type and the two groups are treated as separate populations for purposes of analysis. Target sample sizes and power calculations are presented in the Inception Report, and the achieved sample is presented in Exhibit 5 in Section 4 of this report; Exhibit 47 below disaggregates the quantitative student sample by standard/form. Sampling targets were met for primary and secondary schools and exceeded for students. The Stage 1 sample identified alternate schools (“reserve units”) for use in case of a sampling “refusal,” which occurs when data cannot be collected from a selected school for any reason (e.g., school closure, inaccessibility, non-consent); there was only one Stage 1 refusal among the 30 schools targeted, reflecting a strong Stage 1 response rate of more than 96 percent. February 2018 | ASPIRE Performance Evaluation Report 105 Exhibit 47: Student Sample by Grade Primary Secondary Sex Standard 5 Standard 6 Form 1 Form 2 Total Male 145 148 98 92 483 Female 142 145 96 95 478 Total 287 293 194 187 961 Grand Total 580 381 961 All inferential analysis presented in this report used sampling (probability) weights. Final sampling weights were calculated as the inverse of the ultimate probability of selection for each learner, taking into account all sampling stages. The probability of selection of a learner is calculated as follows: 𝑝(𝑙𝑒𝑎𝑟𝑛𝑒𝑟) = 𝑛ℎ 𝑀ℎ𝑖 𝑀ℎ 𝑚ℎ𝑖 𝑀′ℎ𝑖 Where: 𝑛ℎ = Number of schools to be chosen per stratum 𝑀ℎ𝑖 = Total number of learners in the target grades in the sample school, as shown in the frame 𝑀ℎ = Total number of learners in the target grades in the stratum 𝑚ℎ𝑖 = Total number of students selected in the sampled school 𝑀′ℎ𝑖 = Total number of students observed in the target grades in the sample school on the day of data collection. Due to the time difference between reporting of annual school enrollment data contained in the sampling frame and the data collection for this evaluation, in most cases 𝑀′ℎ𝑖 does not equal 𝑀ℎ𝑖. The final weighting factor is the inverse of the probability of selection, that is: 𝑊(𝑙𝑒𝑎𝑟𝑛𝑒𝑟) = 1 𝑛ℎ 𝑀ℎ 𝑀ℎ𝑖 𝑀′ℎ𝑖 𝑚ℎ𝑖 The formula reflects the two-stage sample design described above. 2.3.2 QUALITATIVE SAMPLE DESIGN The qualitative sample reflected a purposive two-stage design, with schools selected at the first stage, and respondents selected from within these schools and each school’s catchment area at the second stage. Both stages of qualitative sampling reflected a purposive strategy, designed to capture the breadth of school types and stakeholders with the potential to influence ASPIRE’s results. The schools selected for qualitative data collection were drawn from the final quantitative Stage 1 sample. Prior to qualitative sampling, schools in the quantitative sample were classified by four February 2018 | ASPIRE Performance Evaluation Report 106 purposive categories: district, grade level (primary or secondary), school setting (rural or urban), and school performance (categorized as high/low). Because quantitative data collection could not be completed before qualitative data collection, as originally intended, ASPIRE staff assisted in classifying schools as high- or low-performing. The inclusion of high- and low-performing schools facilitated identifying best practices and significant accomplishments (Question 5), implementation challenges (Question 6), and barriers and enablers of success. From this frame, 20 schools were ultimately selected, ensuring representation of each of these categories in each district, in order to maximize potential breadth of schools and capture the diversity of beneficiaries’ experiences. For the second stage of qualitative sampling, the evaluation team identified respondent categories through a stakeholder analysis during inception. Respondent categories represent each level of the ASPIRE’s theory of change, including students (Standards 5 and 6 and Forms 1 and 2); school staff (teachers, head teachers, and teacher trainers); community groups (PTAs, WASH committees, Youth-Friendly Health Service providers, HIV coordinators, and initiation counselors); mothers’ groups; government officials (central- and district-level, including DEMs, coordinating primary education advisors, primary education advisors, and senior education methods advisors); NGO and development partners (ASPIRE implementing partners working as subcontractors to Save the Children and other DREAMS and USAID activities ASPIRE coordinates with); private sector partners; and project-level actors (ASPIRE staff and USAID/Malawi). The evaluation team distributed these categories proportionally across schools to ensure that the final qualitative sample included a suitable number of respondents from each category. The target qualitative sample (total number of focus groups and key informant interviews) was designed to approximate saturation; the final sample, with more than 100 interviews and focus groups, exceeded the target qualitative sample and is presented in Exhibit 6. 2.4 ANALYSIS Data analysis was guided by the ASPIRE’s theory of change, results framework, and evaluation questions. The evaluation team analyzed quantitative data in Stata version 14, first using descriptive statistics (frequency distributions, cross-tabulations) then proceeding to inferential techniques. Analysis gave particular attention to group distributions to identify differences between key subpopulations, and disaggregated all data by sex to compare similarities and differences between males and females. The team examined the distribution of all continuous variables from the reading assessment for normality to inform selection of the correct measures of central tendency (mean versus median) and conducted zero score analysis. All differences that are statistically significant at the 5-percent level are noted as such; tests that yielded insignificant differences are similarly noted. Where the text notes neither significance nor insignificance, it means tests were not performed due to unsuitability. Although Annex 3: Data Tables February 2018 | ASPIRE Performance Evaluation Report 107 presents data disaggregated by district and school setting (urban or rural), these disaggregations are calculated using descriptive techniques due to the sampling limitations described in Section 2.5.3; thus, these results are for informational purposes only and should not be assumed to be reflective of the broader population. In general, the report disaggregates data by sex even where differences are not significant. Unless specifically noted, differences should not be construed as statistically significant. The evaluation team analyzed qualitative data using Dedoose, a cross-platform application that allows collaborative coding by multiple team members and assessment of inter-coder reliability. Qualitative analysis used a two-stage approach to facilitate thematic analysis that addressed each evaluation question and provided insight into the elements underlying ASPIRE’s theory of change. Prior to analysis, the team created and piloted a codebook to ensure relevance of the coding structure to the data and consistent code application by the analysts. The team then refined the codebook and assessed inter-coder reliability by having each team member involved in qualitative analysis code the same document. Code application was compared and discussed to ensure high reliability. This coding structure was applied to all transcripts during the first stage of coding, reflecting a deductive approach to organizing data. The codebook allowed for identification of emerging themes to identify ASPIRE’s contributions to improving achievement in upper primary and secondary school. In the second stage, the evaluation team analyzed each code to generate emergent themes through an inductive process; inductive analysis avoids presupposing hypotheses regarding respondents’ experiences and, thus, allows unexpected results to surface based on respondents’ most salient points. The desk review data extraction guides included built-in coding structures to facilitate a systematic review. Quantitative data were extracted into Microsoft Excel for analysis of trends over ASPIRE’s history, and to prepare for triangulation with primary-source quantitative and qualitative data. Following this analysis of each data type, the evaluation team used a collaborative process to compare emergent themes, triangulate data across sources, and synthesize findings. This approach allowed the evaluation team to view the data from different perspectives and capture new learning. 2.5 LIMITATIONS This performance evaluation has three limitations related to the suitability of baseline data, the data sources and tools, and the sampling design. 2.5.1 LACK OF COMPARABLE BASELINE Although ASPIRE conducted an internal baseline in 2015, before launching activities, baseline data are not suitable for comparison to 2017 performance evaluation data for a number of reasons. As a result, this performance evaluation is limited in that it cannot assess change over time for quantitative indicators. The following specific factors limit the ability to compare baseline and performance evaluation data: February 2018 | ASPIRE Performance Evaluation Report 108  There is no baseline data for Zomba district because ASPIRE did not originally include this district. Furthermore, there is only baseline data for upper primary students; there is no baseline data for secondary school students. As a result, the range of possible comparisons is limited to upper primary students in Balaka and Machinga districts only.  During inception, EnCompass and USAID agreed that revisions to the baseline tools were important to improve data validity and relevance for this performance evaluation. As a result, several variables that exist for both time periods are not comparable.  Many variables in the baseline dataset lack variable and value labels; as a result, it is unclear what the underlying data refer to, resulting in a significant amount of unusable data. Calculations used to produce reading assessment analytic variables are not specified, nor are the original raw data needed to recalculate reading scores available in the dataset. Additionally, because question numbers as given by variable names in the dataset do not align with question numbers in the final version of the baseline tools, the dataset cannot be corrected to include the labels necessary to recover these data.  Baseline data were not weighted, limiting the ability to statistically compare results at the two time periods for even those variables where comparable baseline data do exist. Documentation of the baseline sampling design limits the ability to weight the data retroactively (post-hoc weighting).  Baseline tools lack enumerator instructions and no documentation was available regarding data collector training, which made it difficult to ensure consistent (reliable) measurement at the later period.  Baseline tools exist only in English, even though enumerators were trained and instructed to administer in Chichewa, where appropriate. This made it impossible to verify consistent (reliable) question administration across both periods. Because of these limitations, where baseline data are presented in the main body of the report, they are offset in gray text boxes and comparison to baseline data is not used to draw conclusions regarding ASPIRE’s results. 2.5.2 LIMITATIONS OF DATA SOURCES AND TOOLS  Administrative data from the EMIS and DEMIS systems, which comprise many of the ASPIRE indicators, are frequently of questionable accuracy and timeliness. Care should be taken in interpreting data from these sources.  The reading assessment tool is based on previous reading assessments used in Malawi, mainly the ASPIRE baseline tool and EGRA instruments used in other USAID activities. There is no record of piloting and psychometric analysis of item validity for the former instrument, while the latter was not originally intended for Standards 5 and 6. It was beyond the scope of this evaluation to pilot and assess the psychometric properties of the reading assessment February 2018 | ASPIRE Performance Evaluation Report 109 instrument. Additionally, a lack of grade-level reading benchmarks for Chichewa and English in the upper primary grades means the reading assessment development was not guided by standards agreed upon by experts. Chichewa and English were not designed to be comparable. Given these limitations, reading data should not be compared across the two languages, and consideration should be given when interpreting results as to whether passages are an accurate reflection of grade-level expectations. 2.5.3 LIMITATIONS OF SAMPLING DESIGN  The quantitative sample provides reasonable specificity for estimating health and literacy results among ASPIRE beneficiaries given the evaluation’s purpose. However, as with any evaluation, a larger sample could provide more specific estimates (i.e., narrower confidence intervals) and detect additional differences between subpopulations.  The quantitative sampling reflects an observational evaluation design featuring a single cross￾section (one point in time); the evaluation lacks a rigorously defined counterfactual or quasi￾experimental comparison data. Consequently, the evaluation cannot statistically attribute results to ASPIRE’s interventions (i.e., it cannot establish causality).  The qualitative sample provides breadth of information across the spectrum of ASPIRE stakeholders and contexts, and elicited enablers and constraints at top- and bottom￾performing schools. Qualitative data are, by design, not intended to be generalizable to the population. February 2018 | ASPIRE Performance Evaluation Report 110 ANNEX 3: DATA TABLES Table 1: English oral reading distribution Percent Words per Minute Standard 5 Standard 6 Aggregate 0 1.75% 1.01% 1.37% 1-10 3.50% 1.35% 2.38% 11-20 5.60% 3.37% 4.44% 21-30 12.60% 4.04% 8.23% 31-40 11.55% 6.73% 9.06% 41-50 15.40% 11.42% 13.35% 51-60 22.05% 21.81% 21.90% 61-70 12.95% 17.77% 15.41% 71-80 6.65% 11.08% 8.89% 81-90 3.50% 8.75% 6.15% 91-100 1.40% 5.04% 3.24% 101-110 1.75% 4.37% 3.06% 111-120 0.70% 1.35% 1.02% 121-130 0.70% 1.69% 1.19% 131-140 0.00% 0.34% 0.17% Table 2: Chichewa oral reading distribution Percent Words Per Minute Standard 5 Standard 6 Aggregate 0 2.44% 0.33% 1.37% 1-10 1.05% 0.66% 0.85% 11-20 3.49% 4.00% 3.75% 21-30 17.77% 12.03% 14.88% 31-40 29.28% 22.74% 26.03% 41-50 29.27% 26.75% 28.09% 51-60 10.48% 17.72% 14.04% 61-70 4.19% 10.66% 7.50% 71-80 1.05% 3.98% 2.38% 81-90 0.70% 0.33% 0.51% 91-94 0.35% 0.67% 0.51% February 2018 | ASPIRE Performance Evaluation Report 111 Table 3: Percentage of students who accurately completed English reading comprehension tasks Percent Correct Standard 5 Standard 6 0.00% 67.52 55.76 33.33% 2.14 7.81 50.00% 8.12 15.24 66.67% 2.14 3.72 100.00% 20.09 17.47 Total 100 100 Table 4: Mean English reading comprehension accuracy Linearized Over Mean Std. Err. [95% Conf. Interval] Standard 5 male 32% 0.0472235 22% 42% female 18% 0.0378306 10% 26% Standard 6 male 34% 0.039045 26% 42% female 23% 0.0451651 14% 33% Table 5: Percentage of students who accurately completed Chichewa reading comprehension tasks Percent Correct Standard 5 Standard 6 0.00% 18.84 12.12 20.00% 0 0.67 25.00% 0.72 1.01 33.33% 6.88 6.06 40.00% 1.81 3.37 50.00% 19.2 17.85 60.00% 3.62 5.05 66.67% 15.58 16.16 75.00% 2.54 6.06 80.00% 2.17 5.72 100.00% 28.62 25.93 Total 100 100 Table 6: Mean Chichewa reading comprehension accuracy Linearized Over Mean Std. Err. [95% Conf. Interval] Standard 5 male 59.98% 0.0375828 51.97% 67.99% female 52.15% 0.0542167 40.59% 63.70% Standard 6 male 65.73% 0.0316596 58.98% 72.48% February 2018 | ASPIRE Performance Evaluation Report 112 female 55.93% 0.0414522 47.09% 64.76% Table 7: Percentage of students who accurately completed English listening comprehension tasks Standard 5 Standard 6 0% 58.04 53.02 20% 19.58 22.15 40% 12.24 11.41 60% 8.04 8.05 80% 2.1 4.36 100% 0 1.01 Total 100 100 Table 8: Mean English listening comprehension accuracy Linearized Over Mean Std. Err. [95% Conf. Interval] Standard 5 male 16.89% 0.0199456 12.64% 21.15% female 11.37% 0.0195996 7.20% 15.55% Standard 6 male 18.89% 0.0195406 14.73% 23.06% female 14.91% 0.0172551 11.23% 18.59% Table 9a: Percentage of students who have heard of HIV and AIDS Proportion Std. Err. [95% Conf. Interval] Lower Upper Primary Male 88.59% 0.0162613 0.8463756 0.9162236 Female 88.41% 0.0201508 0.8338067 0.9206589 Secondary Male 99.28% 0.007197 0.9363499 0.9992198 Female 96.26% 0.0098696 0.9332253 0.9793563 Table 9b: Percentage of students who have heard of HIV/AIDS (disaggregated) Primary (n=580) Proportion Balaka Standard 5 Male 0.8693975 Standard 5 Female 0.942863 Standard 6 Male 0.9286933 Standard 6 Female 0.8998566 Machinga Standard 5 Male 0.9121642 Standard 5 Female 0.8077076 Standard 6 Male 0.9542037 Standard 6 Female 0.9357813 Zomba Standard 5 Male 0.8403428 Standard 5 Female 0.8936176 Standard 6 Male 0.7698843 February 2018 | ASPIRE Performance Evaluation Report 113 Standard 6 Female 0.857732 Secondary (n=381) Proportion Balaka Form 1 Male 1 Form 1 Female 0.9802164 Form 2 Male 1 Form 2 Female 0.9619584 Machinga Form 1 Male 1 Form 1 Female 0.9796178 Form 2 Male 1 Form 2 Female 1 Zomba Form 1 Male 0.9749568 Form 1 Female 0.9795242 Form 2 Male 1 Form 2 Female 0.9177774 Table 10a: Percent of students who indicated whether HIV/AIDS is transmitted through: Primary (n=580)* Proportion Std. Err. [95% Conf. Interval] Sexual Intercourse Male 86.18% 0.0207382 81.14% 90.04% Female 83.70% 0.0300809 76.25% 89.15% Contaminated Blood Male 64.85% 0.0362268 56.80% 72.13% Female 51.32% 0.0352439 43.84% 58.75% Mother-to-Child Male 8.26% 0.0146194 5.63% 11.96% Female 11.29% 0.0240648 7.08% 17.52% Other Male 3.90% 0.0125427 1.95% 7.65% Female 4.86% 0.0092747 3.22% 7.26% Secondary (n=381)** Sexual Intercourse Male 98.23% 0.0114777 92.72% 99.59% Female 95.32% 0.011768 91.88% 97.34% Contaminated Blood Male 93.23% 0.0148949 89.06% 95.88% Female 86.84% 0.0255224 80.04% 91.56% Mother-to-Child Male 31.47% 0.032857 24.65% 39.21% Female 34.30% 0.0779848 19.45% 53.03% Other Male 5.01% 0.0261674 1.53% 15.22% Female 3.31% 0.0127459 1.39% 7.68% *Note: 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Table 9, above). February 2018 | ASPIRE Performance Evaluation Report 114 Table 10b: Percentage of students who indicated whether HIV/AIDS is transmitted through (disaggregated): Primary (n=580) Proportion Sexual Intercourse Balaka Standard 5 Male 0.823171 Standard 5 Female 0.902198 Standard 6 Male 0.928693 Standard 6 Female 0.888006 Machinga Standard 5 Male 0.891483 Standard 5 Female 0.744274 Standard 6 Male 0.934053 Standard 6 Female 0.86391 Zomba Standard 5 Male 0.779067 Standard 5 Female 0.847232 Standard 6 Male 0.769884 Standard 6 Female 0.829159 Contaminated Blood Balaka Standard 5 Male 0.648113 Standard 5 Female 0.517874 Standard 6 Male 0.783915 Standard 6 Female 0.679824 Machinga Standard 5 Male 0.68916 Standard 5 Female 0.496865 Standard 6 Male 0.741332 Standard 6 Female 0.491045 Zomba Standard 5 Male 0.487957 Standard 5 Female 0.429724 Standard 6 Male 0.501185 Standard 6 Female 0.521269 Mother-to-Child Balaka Standard 5 Male 0.130437 Standard 5 Female 0.228347 Standard 6 Male 0.152682 Standard 6 Female 0.072428 Machinga Standard 5 Male 0.031948 Standard 5 Female 0.045195 Standard 6 Male 0.107979 Standard 6 Female 0.155251 Zomba Standard 5 Male 0.106381 Standard 5 Female 0.109603 Standard 6 Male 0 Standard 6 Female 0.098834 Other Balaka Standard 5 Male 0.020191 Standard 5 Female 0.058108 Standard 6 Male 0.01967 Standard 6 Female 0.034462 Machinga Standard 5 Male 0.02632 Standard 5 Female 0 February 2018 | ASPIRE Performance Evaluation Report 115 Standard 6 Male 0.025645 Standard 6 Female 0.065211 Zomba Standard 5 Male 0.140007 Standard 5 Female 0.080926 Standard 6 Male 0.007733 Standard 6 Female 0.059198 Secondary (n=381) Proportion Sexual Intercourse Balaka Form 1 Male 0.923509 Form 1 Female 0.980216 Form 2 Male 1 Form 2 Female 0.924867 Machinga Form 1 Male 0.986335 Form 1 Female 0.979618 Form 2 Male 1 Form 2 Female 1 Zomba Form 1 Male 0.974957 Form 1 Female 0.979524 Form 2 Male 1 Form 2 Female 0.896874 Contaminated Blood Balaka Form 1 Male 0.936725 Form 1 Female 0.904133 Form 2 Male 0.975505 Form 2 Female 0.894784 Machinga Form 1 Male 0.822895 Form 1 Female 0.938853 Form 2 Male 0.957075 Form 2 Female 0.953986 Zomba Form 1 Male 0.940451 Form 1 Female 0.830539 Form 2 Male 0.946636 Form 2 Female 0.823414 Mother-to-Child Balaka Form 1 Male 0.373995 Form 1 Female 0.365651 Form 2 Male 0.439332 Form 2 Female 0.340384 Machinga Form 1 Male 0.252541 Form 1 Female 0.322375 Form 2 Male 0.385415 Form 2 Female 0.423105 Zomba Form 1 Male 0.258739 Form 1 Female 0.299574 Form 2 Male 0.287253 Form 2 Female 0.356755 Other Balaka Form 1 Male 0 Form 1 Female 0.075133 Form 2 Male 0.076087 February 2018 | ASPIRE Performance Evaluation Report 116 Form 2 Female 0.037092 Machinga Form 1 Male 0.02865 Form 1 Female 0 Form 2 Male 0.013665 Form 2 Female 0.133703 Zomba Form 1 Male 0.089464 Form 1 Female 0.023985 Form 2 Male 0.045014 Form 2 Female 0 Table 11a: Percent of students who stated it is possible to reduce risk of HIV transmission by: Primary (n=580)* Proportion Std. Err. [95% Conf. Interval] Using a condom Male 65.11% 0.0272974 59.10% 70.69% Female 58.36% 0.0453398 48.50% 67.59% Abstaining from sex Male 64.34% 0.0221408 59.49% 68.91% Female 64.45% 0.0481326 53.67% 73.94% Limiting sexual partners to one Male 9.22% 0.0259488 4.99% 16.44% Female 5.91% 0.0128714 3.69% 9.33% Circumcision Male 6.65% 0.0146998 4.13% 10.56% Female 5.78% 0.0132025 3.53% 9.32% Other Male 8.13% 0.0221446 4.49% 14.27% Female 5.44% 0.0177056 2.69% 10.70% Secondary (n=381)** Proportion Std. Err. [95% Conf. Interval] Using a condom Male 88.72% 0.025802 81.57% 93.32% Female 81.17% 0.0402767 70.56% 88.58% Abstaining from sex Male 85.79% 0.0416624 73.82% 92.82% Female 86.27% 0.0190591 81.45% 89.99% Limiting sexual partners to one Male 35.66% 0.0439482 26.56% 45.93% Female 30.45% 0.056284 19.49% 44.18% Circumcision Male 5.04% 0.0197063 2.08% 11.73% Female 7.71% 0.0259024 3.58% 15.83% Other Male 7.56% 0.0308795 2.96% 17.96% Female 8.70% 0.0317309 3.76% 18.83% *Note: 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Table 9, above). February 2018 | ASPIRE Performance Evaluation Report 117 Table 11b: Percent of students who stated it is possible to reduce risk of HIV transmission by (disaggregated): Primary (n=580) Proportion Using a condom Balaka Standard 5 Male 0.599441 Standard 5 Female 0.787158 Standard 6 Male 0.642859 Standard 6 Female 0.745045 Machinga Standard 5 Male 0.658905 Standard 5 Female 0.616001 Standard 6 Male 0.78754 Standard 6 Female 0.652599 Zomba Standard 5 Male 0.650785 Standard 5 Female 0.513432 Standard 6 Male 0.485417 Standard 6 Female 0.300869 Abstaining from sex Balaka Standard 5 Male 0.611076 Standard 5 Female 0.723333 Standard 6 Male 0.826592 Standard 6 Female 0.730292 Machinga Standard 5 Male 0.575442 Standard 5 Female 0.556893 Standard 6 Male 0.674587 Standard 6 Female 0.709102 Zomba Standard 5 Male 0.605803 Standard 5 Female 0.566956 Standard 6 Male 0.609541 Standard 6 Female 0.636902 Limiting sexual partners to one Balaka Standard 5 Male 0.129876 Standard 5 Female 0.051687 Standard 6 Male 0.113296 Standard 6 Female 0.113159 Machinga Standard 5 Male 0.047678 Standard 5 Female 0.017341 Standard 6 Male 0.178172 Standard 6 Female 0.140552 Zomba Standard 5 Male 0.054771 Standard 5 Female 0.02462 Standard 6 Male 0.019442 Standard 6 Female 0.016978 Circumcision Balaka Standard 5 Male 0.089519 Standard 5 Female 0.136656 Standard 6 Male 0.08113 Standard 6 Female 0.0336 Machinga Standard 5 Male 0.062467 Standard 5 Female 0.024829 February 2018 | ASPIRE Performance Evaluation Report 118 Standard 6 Male 0.056689 Standard 6 Female 0.022108 Zomba Standard 5 Male 0.048232 Standard 5 Female 0.050515 Standard 6 Male 0.07511 Standard 6 Female 0.107196 Secondary (n=381) Proportion Using a condom Balaka Form 1 Male 0.871871 Form 1 Female 0.796 Form 2 Male 0.975505 Form 2 Female 0.807825 Machinga Form 1 Male 0.71024 Form 1 Female 0.89927 Form 2 Male 0.83524 Form 2 Female 0.871365 Zomba Form 1 Male 0.919998 Form 1 Female 0.888696 Form 2 Male 0.940648 Form 2 Female 0.679768 Abstaining from sex Balaka Form 1 Male 0.936646 Form 1 Female 0.900266 Form 2 Male 0.814734 Form 2 Female 0.800817 Machinga Form 1 Male 0.6718 Form 1 Female 0.904519 Form 2 Male 0.792315 Form 2 Female 0.974368 Zomba Form 1 Male 0.919357 Form 1 Female 0.834048 Form 2 Male 0.908298 Form 2 Female 0.837501 Limiting sexual partners to one Balaka Form 1 Male 0.344561 Form 1 Female 0.2519 Form 2 Male 0.254143 Form 2 Female 0.383909 Machinga Form 1 Male 0.32952 Form 1 Female 0.461327 Form 2 Male 0.561285 Form 2 Female 0.437058 Zomba Form 1 Male 0.43013 Form 1 Female 0.217431 Form 2 Male 0.215925 Form 2 Female 0.267158 Circumcision Balaka Form 1 Male 0.1016 Form 1 Female 0.030083 Form 2 Male 0.061807 February 2018 | ASPIRE Performance Evaluation Report 119 Form 2 Female 0.037092 Machinga Form 1 Male 0 Form 1 Female 0.075099 Form 2 Male 0.120515 Form 2 Female 0.127543 Zomba Form 1 Male 0.020453 Form 1 Female 0.020476 Form 2 Male 0.045851 Form 2 Female 0.143646 Table 12a: Percent of students who correctly rejected common misconceptions that it is possible to transmit HIV/AIDS by: Primary (n=580)* Proportion Std. Err. [95% Conf. Interval] Mosquito bites Male 72.13% 0.0271673 65.99% 77.54% Female 73.77% 0.0293784 67.05% 79.54% Sharing clothes/bedding Male 80.08% 0.023 74.72% 84.53% Female 80.12% 0.0215424 75.13% 84.32% Drinking from the same glass Male 73.42% 0.0283976 66.95% 79.03% Female 76.11% 0.018703 71.90% 79.86% Holding hands with someone with HIV or AIDS Male 81.65% 0.0156424 78.08% 84.75% Female 81.26% 0.0273999 74.71% 86.42% Shaking hands with someone with HIV or AIDS Male 82.45% 0.0167479 78.59% 85.74% Female 79.06% 0.025088 73.22% 83.91% Sharing soap with someone with HIV or AIDS Male 77.81% 0.0225336 72.64% 82.24% Female 75.03% 0.0203978 70.44% 79.13% Bathing in the same water as someone with HIV or AIDS Male 76.80% 0.0229968 71.55% 81.34% Female 78.26% 0.0209526 73.46% 82.39% Witchcraft Male 76.03% 0.0230486 70.78% 80.60% Female 74.33% 0.0203251 69.77% 78.42% Is it possible to cure AIDS? No 68.62 February 2018 | ASPIRE Performance Evaluation Report 120 Secondary (n=381)** Proportion Std. Err. [95% Conf. Interval] Mosquito bites Male 92.71% 0.0217736 86.12% 96.31% Female 88.14% 0.024388 81.55% 92.59% Sharing clothes/bedding Male 98.25% 0.0110834 93.04% 99.58% Female 93.81% 0.0087244 91.56% 95.50% Drinking from the same glass Male 97.01% 0.0116086 93.01% 98.75% Female 93.99% 0.0028607 93.32% 94.60% Holding hands with someone with HIV or AIDS Male 96.91% 0.012241 92.65% 98.73% Female 95.56% 0.0098041 92.79% 97.30% Shaking hands with someone with HIV or AIDS Male 96.24% 0.0129485 92.02% 98.27% Female 94.92% 0.0113633 91.70% 96.93% Sharing soap with someone with HIV or AIDS Male 97.22% 0.0122013 92.75% 98.97% Female 89.40% 0.0154725 85.43% 92.39% Bathing in the same water as someone with HIV or AIDS Male 96.02% 0.0209058 87.70% 98.79% Female 94.86% 0.0099035 92.15% 96.67% Witchcraft Male 98.17% 0.0094377 94.33% 99.42% Female 89.49% 0.0173143 84.96% 92.77% Is it possible to cure AIDS? No 85.04 *Note: 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Table 9, above). Table 12b: Percentage of students who correctly rejected common misconceptions that it is possible to transmit HIV transmission by (disaggregated): Primary (n=580) Proportion Mosquito bites Balaka Standard 5 Male 0.625873 Standard 5 Female 0.827179 Standard 6 Male 0.872973 Standard 6 Female 0.723924 Machinga Standard 5 Male 0.725236 Standard 5 Female 0.667171 Standard 6 Male 0.814176 Standard 6 Female 0.676504 Zomba Standard 5 Male 0.638554 Standard 5 Female 0.79849 Standard 6 Male 0.611958 Standard 6 Female 0.783154 Sharing clothes/bedding Balaka Standard 5 Male 0.829015 Standard 5 Female 0.874144 Standard 6 Male 0.853258 Standard 6 Female 0.860156 February 2018 | ASPIRE Performance Evaluation Report 121 Machinga Standard 5 Male 0.833825 Standard 5 Female 0.680846 Standard 6 Male 0.850897 Standard 6 Female 0.862539 Zomba Standard 5 Male 0.777748 Standard 5 Female 0.754675 Standard 6 Male 0.636558 Standard 6 Female 0.827499 Drinking from the same glass Balaka Standard 5 Male 0.693422 Standard 5 Female 0.849456 Standard 6 Male 0.811082 Standard 6 Female 0.787944 Machinga Standard 5 Male 0.761042 Standard 5 Female 0.584533 Standard 6 Male 0.836229 Standard 6 Female 0.872237 Zomba Standard 5 Male 0.724066 Standard 5 Female 0.835104 Standard 6 Male 0.524334 Standard 6 Female 0.698158 Holding hands with someone with HIV or AIDS Balaka Standard 5 Male 0.782299 Standard 5 Female 0.862005 Standard 6 Male 0.892643 Standard 6 Female 0.832305 Machinga Standard 5 Male 0.82895 Standard 5 Female 0.6887 Standard 6 Male 0.884348 Standard 6 Female 0.8305 Zomba Standard 5 Male 0.805133 Standard 5 Female 0.861931 Standard 6 Male 0.674411 Standard 6 Female 0.844864 Shaking hands with someone with HIV or AIDS Balaka Standard 5 Male 0.809267 Standard 5 Female 0.838313 Standard 6 Male 0.892643 Standard 6 Female 0.832305 Machinga Standard 5 Male 0.812808 Standard 5 Female 0.656987 Standard 6 Male 0.904499 Standard 6 Female 0.8305 Zomba Standard 5 Male 0.805133 Standard 5 Female 0.861931 Standard 6 Male 0.698894 Standard 6 Female 0.77672 February 2018 | ASPIRE Performance Evaluation Report 122 Sharing soap with someone with HIV or AIDS Balaka Standard 5 Male 0.725546 Standard 5 Female 0.815563 Standard 6 Male 0.908978 Standard 6 Female 0.814933 Machinga Standard 5 Male 0.779843 Standard 5 Female 0.608996 Standard 6 Male 0.870865 Standard 6 Female 0.812776 Zomba Standard 5 Male 0.751681 Standard 5 Female 0.715425 Standard 6 Male 0.599301 Standard 6 Female 0.790045 Bathing in the same water as someone with HIV or AIDS Balaka Standard 5 Male 0.731834 Standard 5 Female 0.827701 Standard 6 Male 0.872033 Standard 6 Female 0.765333 Machinga Standard 5 Male 0.78907 Standard 5 Female 0.670195 Standard 6 Male 0.825996 Standard 6 Female 0.824372 Zomba Standard 5 Male 0.752974 Standard 5 Female 0.844954 Standard 6 Male 0.610893 Standard 6 Female 0.790045 Witchcraft Balaka Standard 5 Male 0.67407 Standard 5 Female 0.781381 Standard 6 Male 0.871602 Standard 6 Female 0.870515 Machinga Standard 5 Male 0.779145 Standard 5 Female 0.606761 Standard 6 Male 0.842653 Standard 6 Female 0.763386 Zomba Standard 5 Male 0.694847 Standard 5 Female 0.745193 Standard 6 Male 0.653574 Standard 6 Female 0.768231 February 2018 | ASPIRE Performance Evaluation Report 123 Secondary (n=381) Proportion Mosquito bites Balaka Form 1 Male 0.949863 Form 1 Female 0.813884 Form 2 Male 0.975505 Form 2 Female 0.922391 Machinga Form 1 Male 0.87439 Form 1 Female 0.825281 Form 2 Male 0.888665 Form 2 Female 0.825099 Zomba Form 1 Male 0.974957 Form 1 Female 0.898985 Form 2 Male 0.881616 Form 2 Female 0.917777 Sharing clothes/bedding Balaka Form 1 Male 0.974892 Form 1 Female 0.980216 Form 2 Male 1 Form 2 Female 0.924867 Machinga Form 1 Male 1 Form 1 Female 0.910949 Form 2 Male 0.9427 Form 2 Female 0.948737 Zomba Form 1 Male 0.974957 Form 1 Female 0.955539 Form 2 Male 1 Form 2 Female 0.917777 Drinking from the same glass Balaka Form 1 Male 1 Form 1 Female 0.980216 Form 2 Male 1 Form 2 Female 0.924867 Machinga Form 1 Male 0.986335 Form 1 Female 0.945283 Form 2 Male 0.917925 Form 2 Female 0.929717 Zomba Form 1 Male 0.974957 Form 1 Female 0.955539 Form 2 Male 0.954149 Form 2 Female 0.917777 Holding hands with someone with HIV or AIDS Balaka Form 1 Male 0.974892 Form 1 Female 0.980216 Form 2 Male 1 Form 2 Female 0.961958 Machinga Form 1 Male 1 Form 1 Female 0.945283 Form 2 Male 0.917925 Form 2 Female 0.974368 February 2018 | ASPIRE Performance Evaluation Report 124 Zomba Form 1 Male 0.974957 Form 1 Female 0.979524 Form 2 Male 0.954149 Form 2 Female 0.917777 Shaking hands with someone with HIV or AIDS Balaka Form 1 Male 0.949784 Form 1 Female 0.980216 Form 2 Male 1 Form 2 Female 0.961958 Machinga Form 1 Male 0.97074 Form 1 Female 0.934966 Form 2 Male 0.917925 Form 2 Female 0.929717 Zomba Form 1 Male 0.974957 Form 1 Female 0.979524 Form 2 Male 0.954149 Form 2 Female 0.917777 Sharing soap with someone with HIV or AIDS Balaka Form 1 Male 0.974892 Form 1 Female 0.950133 Form 2 Male 1 Form 2 Female 0.887775 Machinga Form 1 Male 0.97074 Form 1 Female 0.924901 Form 2 Male 0.97135 Form 2 Female 0.859433 Zomba Form 1 Male 0.974957 Form 1 Female 0.85643 Form 2 Male 0.954149 Form 2 Female 0.917777 Bathing in the same water as someone with HIV or AIDS Balaka Form 1 Male 0.949784 Form 1 Female 0.950133 Form 2 Male 0.975582 Form 2 Female 0.961958 Machinga Form 1 Male 1 Form 1 Female 0.979618 Form 2 Male 0.97135 Form 2 Female 0.904085 Zomba Form 1 Male 0.974957 Form 1 Female 0.979524 Form 2 Male 0.908298 Form 2 Female 0.917777 Witchcraft Balaka Form 1 Male 0.935146 Form 1 Female 0.922391 Form 2 Male 1 Form 2 Female 0.82995 February 2018 | ASPIRE Performance Evaluation Report 125 Machinga Form 1 Male 1 Form 1 Female 0.919652 Form 2 Male 0.97135 Form 2 Female 0.885065 Zomba Form 1 Male 0.974957 Form 1 Female 0.912681 Form 2 Male 1 Form 2 Female 0.882891 Table 13a: Where students indicated having heard of HIV/AIDS Primary (n=580)* Proportion Std. Err. [95% Conf. Interval] Hospital Male 33.73% 0.04038 25.73% 42.80% Female 31.31% 0.047137 22.23% 42.11% Radio Male 29.62% 0.03551 22.65% 37.70% Female 35.78% 0.062769 23.73% 49.93% Teacher Male 71.08% 0.030842 64.10% 77.19% Female 69.01% 0.03077 62.11% 75.16% Video/Film Male 4.33% 0.00931 2.73% 6.81% Female 1.52% 0.009589 0.39% 5.70% Drama/Songs Male 7.02% 0.023868 3.35% 14.14% Female 7.06% 0.028239 2.95% 15.97% Friends Male 31.86% 0.038682 24.23% 40.60% Female 36.51% 0.060763 24.75% 50.14% Relatives Male 27.41% 0.043315 19.19% 37.52% Female 41.50% 0.047655 31.83% 51.88% Television Male 4.06% 0.012827 2.05% 7.86% Female 3.68% 0.013093 1.71% 7.75% Other Male 8.66% 0.018279 5.48% 13.43% Female 4.58% 0.014602 2.30% 8.91% Secondary (n=381)** Proportion Std. Err. [95% Conf. Interval] Hospital Male 49.22% 0.053302 37.60% 60.92% Female 49.29% 0.035279 41.51% 57.11% Radio Male 40.97% 0.04972 30.51% 52.32% Female 39.28% 0.05191 28.49% 51.24% Teacher Male 84.68% 0.030148 76.71% 90.27% Female 84.88% 0.024235 78.66% 89.53% Video/Film Male 6.89% 0.018932 3.69% 12.50% Female 4.78% 0.021916 1.69% 12.80% February 2018 | ASPIRE Performance Evaluation Report 126 Drama/Songs Male 13.52% 0.034001 7.56% 23.01% Female 15.27% 0.040819 8.20% 26.69% Friends Male 35.33% 0.038081 27.37% 44.20% Female 32.60% 0.07321 18.71% 50.40% Relatives Male 38.80% 0.043349 29.68% 48.77% Female 42.61% 0.038921 34.25% 51.42% Television Male 7.52% 0.011894 5.27% 10.64% Female 7.54% 0.022399 3.84% 14.30% Other Male 21.19% 0.037073 14.08% 30.60% Female 14.98% 0.025205 10.18% 21.50% *Note: 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Table 9, above). Table 13b: Where students indicated having heard of HIV/AIDS (disaggregated) Primary (n=580) Proportion Hospital Balaka Standard 5 Male 0.298164 Standard 5 Female 0.453677 Standard 6 Male 0.395503 Standard 6 Female 0.454687 Machinga Standard 5 Male 0.379787 Standard 5 Female 0.315931 Standard 6 Male 0.395169 Standard 6 Female 0.481287 Zomba Standard 5 Male 0.193733 Standard 5 Female 0.122304 Standard 6 Male 0.315946 Standard 6 Female 0.115439 Radio Balaka Standard 5 Male 0.195434 Standard 5 Female 0.393041 Standard 6 Male 0.347904 Standard 6 Female 0.365135 Machinga Standard 5 Male 0.252986 Standard 5 Female 0.305299 Standard 6 Male 0.341479 Standard 6 Female 0.401159 Zomba Standard 5 Male 0.376462 Standard 5 Female 0.287573 Standard 6 Male 0.249915 Standard 6 Female 0.40537 Teacher Balaka Standard 5 Male 0.622758 Standard 5 Female 0.760519 Standard 6 Male 0.811559 Standard 6 Female 0.73513 February 2018 | ASPIRE Performance Evaluation Report 127 Machinga Standard 5 Male 0.728708 Standard 5 Female 0.622473 Standard 6 Male 0.824596 Standard 6 Female 0.823269 Zomba Standard 5 Male 0.682735 Standard 5 Female 0.613954 Standard 6 Male 0.531949 Standard 6 Female 0.614368 Video/Film Balaka Standard 5 Male 0.066465 Standard 5 Female 0 Standard 6 Male 0.019239 Standard 6 Female 0 Machinga Standard 5 Male 0.016483 Standard 5 Female 0.020366 Standard 6 Male 0.085583 Standard 6 Female 0.038167 Zomba Standard 5 Male 0.007824 Standard 5 Female 0 Standard 6 Male 0.055634 Standard 6 Female 0.016978 Drama/Songs Balaka Standard 5 Male 0.112202 Standard 5 Female 0.095581 Standard 6 Male 0.176732 Standard 6 Female 0.065682 Machinga Standard 5 Male 0.062533 Standard 5 Female 0.061099 Standard 6 Male 0.02081 Standard 6 Female 0.136609 Zomba Standard 5 Male 0.082531 Standard 5 Female 0 Standard 6 Male 0.023198 Standard 6 Female 0.059263 Friends Balaka Standard 5 Male 0.364675 Standard 5 Female 0.397753 Standard 6 Male 0.357535 Standard 6 Female 0.455431 Machinga Standard 5 Male 0.331091 Standard 5 Female 0.323987 Standard 6 Male 0.205011 Standard 6 Female 0.442733 Zomba Standard 5 Male 0.409872 Standard 5 Female 0.357797 Standard 6 Male 0.311126 Standard 6 Female 0.253319 February 2018 | ASPIRE Performance Evaluation Report 128 Relatives Balaka Standard 5 Male 0.347815 Standard 5 Female 0.489003 Standard 6 Male 0.332738 Standard 6 Female 0.479543 Machinga Standard 5 Male 0.283656 Standard 5 Female 0.408843 Standard 6 Male 0.25313 Standard 6 Female 0.523525 Zomba Standard 5 Male 0.244599 Standard 5 Female 0.362653 Standard 6 Male 0.214897 Standard 6 Female 0.260473 Television Balaka Standard 5 Male 0.026479 Standard 5 Female 0.022277 Standard 6 Male 0.019239 Standard 6 Female 0.021749 Machinga Standard 5 Male 0.058268 Standard 5 Female 0.040733 Standard 6 Male 0.067265 Standard 6 Female 0.035075 Zomba Standard 5 Male 0 Standard 5 Female 0.067499 Standard 6 Male 0.043925 Standard 6 Female 0.025467 Secondary (n=381) Proportion Hospital Balaka Form 1 Male 0.219258 Form 1 Female 0.285191 Form 2 Male 0.471285 Form 2 Female 0.437882 Machinga Form 1 Male 0.3855 Form 1 Female 0.578986 Form 2 Male 0.625 Form 2 Female 0.702624 Zomba Form 1 Male 0.579447 Form 1 Female 0.381677 Form 2 Male 0.513159 Form 2 Female 0.571771 Radio Balaka Form 1 Male 0.399182 Form 1 Female 0.547391 Form 2 Male 0.448237 Form 2 Female 0.366091 Machinga Form 1 Male 0.358085 Form 1 Female 0.68731 Form 2 Male 0.467365 Form 2 Female 0.642658 February 2018 | ASPIRE Performance Evaluation Report 129 Zomba Form 1 Male 0.422727 Form 1 Female 0.213922 Form 2 Male 0.374492 Form 2 Female 0.312962 Teacher Balaka Form 1 Male 0.833626 Form 1 Female 0.900266 Form 2 Male 0.93827 Form 2 Female 0.850684 Machinga Form 1 Male 0.83849 Form 1 Female 0.89284 Form 2 Male 0.78844 Form 2 Female 0.883703 Zomba Form 1 Male 0.830175 Form 1 Female 0.820352 Form 2 Male 0.866589 Form 2 Female 0.828633 Video/Film Balaka Form 1 Male 0.089883 Form 1 Female 0.056875 Form 2 Male 0.125 Form 2 Female 0.039567 Machinga Form 1 Male 0 Form 1 Female 0.202624 Form 2 Male 0.11521 Form 2 Female 0.044652 Zomba Form 1 Male 0.06019 Form 1 Female 0 Form 2 Male 0.053364 Form 2 Female 0.034886 Drama/Songs Balaka Form 1 Male 0.245866 Form 1 Female 0.149316 Form 2 Male 0.086225 Form 2 Female 0.068125 Machinga Form 1 Male 0.19402 Form 1 Female 0.247276 Form 2 Male 0.11521 Form 2 Female 0.397907 Zomba Form 1 Male 0.090747 Form 1 Female 0.114711 Form 2 Male 0.139399 Form 2 Female 0.082223 Friends Balaka Form 1 Male 0.33267 Form 1 Female 0.410191 Form 2 Male 0.382056 Form 2 Female 0.278558 February 2018 | ASPIRE Performance Evaluation Report 130 Machinga Form 1 Male 0.34635 Form 1 Female 0.637409 Form 2 Male 0.194645 Form 2 Female 0.519931 Zomba Form 1 Male 0.460595 Form 1 Female 0.146977 Form 2 Male 0.306422 Form 2 Female 0.291749 Relatives Balaka Form 1 Male 0.322453 Form 1 Female 0.43035 Form 2 Male 0.287312 Form 2 Female 0.30535 Machinga Form 1 Male 0.40426 Form 1 Female 0.544652 Form 2 Male 0.368585 Form 2 Female 0.6193 Zomba Form 1 Male 0.429151 Form 1 Female 0.443344 Form 2 Male 0.422295 Form 2 Female 0.319884 Television Balaka Form 1 Male 0.075166 Form 1 Female 0.094917 Form 2 Male 0 Form 2 Female 0.06965 Machinga Form 1 Male 0 Form 1 Female 0.068669 Form 2 Male 0.05791 Form 2 Female 0.188672 Zomba Form 1 Male 0.089801 Form 1 Female 0.023985 Form 2 Male 0.150217 Form 2 Female 0.080277 February 2018 | ASPIRE Performance Evaluation Report 131 Table 14a: Percent of students who have accessed sexual and reproductive health services via the following: Proportion Std. Err. [95% Conf. Interval] Proportion who have visited doctor, nurse, or health clinic for sexual and reproductive health services in current academic year Primary (n=580) Male 0.0525895 0.015429 0.027888 0.096987 Female 0.0498856 0.016201 0.024713 0.098119 Secondary (n=381) Male 0.0882325 0.027634 0.043075 0.172211 Female 0.0646136 0.022002 0.02978 0.134541 Proportion who have used an Youth￾Friendly Health Services in current academic year Primary (n=580) Male 0.1131256 0.018102 0.079894 0.157808 Female 0.0415115 0.012396 0.021808 0.077606 Secondary (n=381) Male 0.1707258 0.037472 0.102457 0.270762 Female 0.1867768 0.059143 0.08797 0.353543 Proportion who have gone for HIV counseling and testing Primary* (n=580) Male 0.3869697 0.052992 0.281669 0.504015 Female 0.3352367 0.036225 0.262877 0.416266 Secondary** (n=381) Male 0.6467308 0.059998 0.504893 0.76671 Female 0.7215211 0.079706 0.517029 0.862463 Proportion who are worried about health problems Primary (n=580) Male 0.9308704 0.017459 0.883073 0.960014 Female 0.9137338 0.013185 0.881173 0.938 Secondary (n=381) Male 0.9288475 0.022767 0.85834 0.965666 Female 0.9665901 0.016642 0.901738 0.989155 *Note: 11.9 percent of primary respondents and 1.9 percent of secondary respondents were not asked this question because they had not heard of HIV/AIDS (Table 9, above). Table 14b: Percentage of students who have accessed sexual and reproductive health services via the following (disaggregated): Primary (n=580) Proportion Proportion who have visited doctor, nurse, or health clinic for sexual and reproductive health services in current academic year Balaka Standard 5 Male 0.06304 Standard 5 Female 0.044999 Standard 6 Male 0.105392 Standard 6 Female 0 Machinga Standard 5 Male 0 Standard 5 Female 0.037341 Standard 6 Male 0.121709 Standard 6 Female 0.101946 Zomba Standard 5 Male 0 Standard 5 Female 0.007564 February 2018 | ASPIRE Performance Evaluation Report 132 Standard 6 Male 0.027065 Standard 6 Female 0.079795 Proportion who have used an Youth-Friendly Health Services in current academic year Balaka Standard 5 Male 0.053375 Standard 5 Female 0.080297 Standard 6 Male 0.243137 Standard 6 Female 0.077667 Machinga Standard 5 Male 0.103555 Standard 5 Female 0.083196 Standard 6 Male 0.141946 Standard 6 Female 0.020817 Zomba Standard 5 Male 0.027386 Standard 5 Female 0 Standard 6 Male 0.110829 Standard 6 Female 0.00682 Proportion who have gone for HIV counseling and testing Balaka Standard 5 Male 0.391296 Standard 5 Female 0.25144 Standard 6 Male 0.466943 Standard 6 Female 0.338894 Machinga Standard 5 Male 0.340555 Standard 5 Female 0.279685 Standard 6 Male 0.416843 Standard 6 Female 0.469425 Zomba Standard 5 Male 0.412795 Standard 5 Female 0.246032 Standard 6 Male 0.301726 Standard 6 Female 0.371496 Secondary (n=381) Proportion Proportion who have visited doctor, nurse, or health clinic for sexual and reproductive health services in current academic year Balaka Form 1 Male 0.063354 Form 1 Female 0 Form 2 Male 0.097825 Form 2 Female 0.068125 Machinga Form 1 Male 0.053425 Form 1 Female 0.16829 Form 2 Male 0.09635 Form 2 Female 0.133703 Zomba Form 1 Male 0.139305 Form 1 Female 0 Form 2 Male 0.045014 Form 2 Female 0.080277 Proportion who have used an Youth-Friendly Health Services in current academic year Balaka Form 1 Male 0.204699 Form 1 Female 0.056875 Form 2 Male 0.160848 Form 2 Female 0.111275 Machinga Form 1 Male 0.178425 Form 1 Female 0.220733 February 2018 | ASPIRE Performance Evaluation Report 133 Form 2 Male 0.302204 Form 2 Female 0.289402 Zomba Form 1 Male 0.124611 Form 1 Female 0.207445 Form 2 Male 0.136716 Form 2 Female 0.176895 Proportion who have gone for HIV counseling and testing Balaka Form 1 Male 0.512594 Form 1 Female 0.531693 Form 2 Male 0.544522 Form 2 Female 0.555758 Machinga Form 1 Male 0.590644 Form 1 Female 0.688758 Form 2 Male 0.770999 Form 2 Female 0.725983 Zomba Form 1 Male 0.656044 Form 1 Female 0.839342 Form 2 Male 0.71674 Form 2 Female 0.727553 Table 15: Satisfaction with Youth-Friendly Health Services referral Primary (n=46) Male Female Total Very unsatisfied 6.06 0 4.35 Unsatisfied 6.06 7.69 6.52 Very Satisfied 57.58 30.77 50 Satisfied 30.3 61.54 39.13 Total 100 100 100 Secondary (n=62) Male Female Total Very unsatisfied 12.5 3.33 8.06 Unsatisfied 3.13 3.33 3.23 Very Satisfied 56.25 66.67 61.29 Satisfied 28.13 26.67 27.42 Total 100 100 100 Table 16a: Proportion who report knowing of contraceptives in general Primary (n=580) Male 0.2960136 0.037792 0.222198 0.382298 Female 0.3385602 0.059214 0.225602 0.473495 Secondary (n=381) Male 0.6702654 0.059038 0.528516 0.786605 Female 0.6549216 0.025767 0.595489 0.709876 February 2018 | ASPIRE Performance Evaluation Report 134 Table 16b: Of those who reported knowing of contraceptive methods, what types did they know about? (Top five) Primary (n=184) Male Female Total Condom 40.23 54.64 47.83 Injectables (Depo) 62.07 70.1 66.3 Pill 28.74 42.27 35.87 IUD 17.24 20.62 19.02 Sterilization 6.9 6.19 6.52 Secondary (n=260) Male Female Total Condom 58.02 51.94 55 Injectables (Depo) 54.2 70.54 62.31 Pill 45.8 58.91 52.31 IUD 50.38 62.02 56.15 Sterilization 34.35 26.36 30.38 Table 16c: Proportion who report knowing of contraceptives in general (disaggregated) Primary (n=580) Proportion Balaka Standard 5 Male 0.2019865 Standard 5 Female 0.3395201 Standard 6 Male 0.4432233 Standard 6 Female 0.4253188 Machinga Standard 5 Male 0.3151043 Standard 5 Female 0.3208181 Standard 6 Male 0.4275555 Standard 6 Female 0.4900715 Zomba Standard 5 Male 0.1343075 Standard 5 Female 0.1862528 Standard 6 Male 0.1881438 Standard 6 Female 0.2712294 Secondary (n=381) Proportion Balaka Form 1 Male 0.7438219 Form 1 Female 0.751391 Form 2 Male 0.8145637 Form 2 Female 0.5070084 Machinga Form 1 Male 0.6673148 Form 1 Female 0.6762632 Form 2 Male 0.8502249 Form 2 Female 0.8992695 Zomba Form 1 Male 0.663151 Form 1 Female 0.6748765 Form 2 Male 0.4621574 February 2018 | ASPIRE Performance Evaluation Report 135 Form 2 Female 0.7438219 Table 16d: Of those who reported knowing of contraceptive methods, what types did they know about? (Top five, disaggregated) Primary (n=580) Proportion Condom Balaka Standard 5 Male 0.554616 Standard 5 Female 0.625777 Standard 6 Male 0.387625 Standard 6 Female 0.667378 Machinga Standard 5 Male 0.299841 Standard 5 Female 0.572542 Standard 6 Male 0.348701 Standard 6 Female 0.497381 Zomba Standard 5 Male 0.44661 Standard 5 Female 0.386197 Standard 6 Male 0.294526 Standard 6 Female 0.048868 Injectables (Depo) Balaka Standard 5 Male 0.670945 Standard 5 Female 0.717829 Standard 6 Male 0.754032 Standard 6 Female 0.694528 Machinga Standard 5 Male 0.551276 Standard 5 Female 0.543852 Standard 6 Male 0.653039 Standard 6 Female 0.782346 Zomba Standard 5 Male 0.466059 Standard 5 Female 0.878173 Standard 6 Male 0.575343 Standard 6 Female 0.925986 Pill Balaka Standard 5 Male 0.17345 Standard 5 Female 0.360999 Standard 6 Male 0.320007 Standard 6 Female 0.573895 Machinga Standard 5 Male 0.271442 Standard 5 Female 0.307155 Standard 6 Male 0.386874 Standard 6 Female 0.483202 Zomba Standard 5 Male 0.262157 Standard 5 Female 0.354769 Standard 6 Male 0.17123 Standard 6 Female 0.387886 IUD Balaka Standard 5 Male 0.131091 Standard 5 Female 0.240753 Standard 6 Male 0.04005 Standard 6 Female 0.337535 February 2018 | ASPIRE Performance Evaluation Report 136 Machinga Standard 5 Male 0.143997 Standard 5 Female 0 Standard 6 Male 0.427019 Standard 6 Female 0.20892 Zomba Standard 5 Male 0 Standard 5 Female 0.170126 Standard 6 Male 0.260262 Standard 6 Female 0.212572 Sterilization Balaka Standard 5 Male 0.131091 Standard 5 Female 0.052767 Standard 6 Male 0.084532 Standard 6 Female 0.143907 Machinga Standard 5 Male 0.117943 Standard 5 Female 0.063482 Standard 6 Male 0.048672 Standard 6 Female 0.042477 Zomba Standard 5 Male 0 Standard 5 Female 0 Standard 6 Male 0 Standard 6 Female 0 Secondary (n=381) Proportion Condom Balaka Form 1 Male 0.398055 Form 1 Female 0.460889 Form 2 Male 0.552076 Form 2 Female 0.305879 Machinga Form 1 Male 0.528635 Form 1 Female 0.555865 Form 2 Male 0.716757 Form 2 Female 0.752825 Zomba Form 1 Male 0.768375 Form 1 Female 0.665102 Form 2 Male 0.546542 Form 2 Female 0.420854 Injectables (Depo) Balaka Form 1 Male 0.536451 Form 1 Female 0.706554 Form 2 Male 0.435384 Form 2 Female 0.787593 Machinga Form 1 Male 0.645842 Form 1 Female 0.757975 Form 2 Male 0.427275 Form 2 Female 0.831183 Zomba Form 1 Male 0.677186 Form 1 Female 0.748901 Form 2 Male 0.577567 Form 2 Female 0.659473 February 2018 | ASPIRE Performance Evaluation Report 137 Pill Balaka Form 1 Male 0.603962 Form 1 Female 0.722292 Form 2 Male 0.497324 Form 2 Female 0.38392 Machinga Form 1 Male 0.438046 Form 1 Female 0.762644 Form 2 Male 0.456414 Form 2 Female 0.787165 Zomba Form 1 Male 0.338593 Form 1 Female 0.546212 Form 2 Male 0.391409 Form 2 Female 0.273314 IUD Balaka Form 1 Male 0.671025 Form 1 Female 0.453214 Form 2 Male 0.48834 Form 2 Female 0.787593 Machinga Form 1 Male 0.459439 Form 1 Female 0.652699 Form 2 Male 0.460137 Form 2 Female 0.563157 Zomba Form 1 Male 0.460199 Form 1 Female 0.694841 Form 2 Male 0.2673 Form 2 Female 0.604685 Sterilization Balaka Form 1 Male 0.346531 Form 1 Female 0.119025 Form 2 Male 0.319453 Form 2 Female 0.150063 Machinga Form 1 Male 0.354158 Form 1 Female 0.364572 Form 2 Male 0.322461 Form 2 Female 0.364323 Zomba Form 1 Male 0.295245 Form 1 Female 0.23363 Form 2 Male 0.453458 Form 2 Female 0.28292 February 2018 | ASPIRE Performance Evaluation Report 138 Table 17: Proportion of students who used a condom at last intercourse Proportion Std. Err. [95% Conf.Interval] Primary (n=97) No because don't know of contraceptives Male 0.523828 0.044522 0.42863 0.617325 Female 0.279957 0.077087 0.146182 0.468918 No Male 0.203102 0.036208 0.136246 0.291685 Female 0.436658 0.120037 0.213938 0.688234 Yes Male 0.273071 0.046421 0.185327 0.382835 Female 0.283386 0.108686 0.11149 0.554816 Secondary (n=117) No because don't know of contraceptives Male 0.199741 0.034609 0.132652 0.289437 Female 0.269178 0.11375 0.090557 0.576702 No Male 0.214199 0.05812 0.110966 0.37316 Female 0.210686 0.045271 0.12602 0.33071 Yes Male 0.58606 0.057076 0.453999 0.706808 Female 0.520137 0.105417 0.294249 0.738081 Table 18: Percent of students who plan to use a condom in the future, out of those sexually active Proportion Std. Err. [95% Conf. Interval] Primary* (n=97) No Male 0.5893512 0.0426604 0.4958054 0.676853 Female 0.4210163 0.0637573 0.2932639 0.5602998 Yes Male 0.4002386 0.040832 0.3166309 0.4900902 Female 0.5789837 0.0637573 0.4397002 0.7067361 Unsure Male 0.0104102 0.0102347 0.0012476 0.0813838 Female 0 No observations Secondary** (n=117) No Male 0.2631718 0.0270359 0.206698 0.3286824 Female 0.4811257 0.1131958 0.2495036 0.7211544 Yes Male 0.7026059 0.0219802 0.6506222 0.7498291 Female 0.4461803 0.1513345 0.1677668 0.7630194 Unsure Male 0.0342223 0.0251592 0.0062927 0.1654728 Female 0.0726941 0.0454457 0.0167724 0.2648441 Table 19: Percentage of girls who come to school when menstruating Proportion Std. Err. [95% Conf. Interval] Primary (n=112) 0.8762 0.034899 0.780943 0.933559 Secondary (n=179) 0.95307 0.01526 0.903719 0.977748 February 2018 | ASPIRE Performance Evaluation Report 139 Table 20: Percentage of students who feel comfortable using school toilets Proportion Std. Err. [95% Conf. Interval] Primary (n=580) Male 86% 0.026926 0.791876 0.907811 Female 86% 0.023786 0.805007 0.907309 Secondary (n=381) Male 91% 0.029631 0.819478 0.958762 Female 92% 0.034748 0.79925 0.967086 Table 21a: Percent of students who plan to use a condom in the future, out of all Proportion Std. Err. [95% Conf. Interval] Primary* (n=187) No Male 0.138003 0.037208 0.075952 0.237707 Female 0.335663 0.044934 0.247465 0.43704 Yes Male 0.823239 0.031714 0.74534 0.88111 Female 0.635396 0.035487 0.556987 0.707222 Unsure Male 0.038759 0.020883 0.012062 0.117514 Female 0.028941 0.027672 0.003641 0.195538 Secondary** (n=259) No Male 0.1555 0.036558 0.089329 0.256863 Female 0.265521 0.043308 0.179488 0.373996 Yes Male 0.821338 0.042111 0.706048 0.897946 Female 0.64211 0.038418 0.551406 0.723664 Unsure Male 0.023162 0.014984 0.005272 0.095898 Female 0.09237 0.025512 0.048641 0.168448 *Primary: Relationship between sex and likelihood to use condom in future significant at 1 percent (p=0.0020; design-based F) **Secondary: Relationship between sex and likelihood to use condom in future significant at 5 percent (p=0.0406; design-based F) Table 21b: Percent of students who plan to use a condom in the future, out of all (disaggregated) Primary (n=580) Proportion Balaka Standard 5 Male 0.5952666 Standard 5 Female 0.7599826 Standard 6 Male 0.7760343 Standard 6 Female 0.7944347 Machinga Standard 5 Male 0.8560027 Standard 5 Female 0.7457567 Standard 6 Male 0.9093011 Standard 6 Female 0.6559331 Zomba Standard 5 Male 1 Standard 5 Female 0.4344954 Standard 6 Male 0.5958863 Standard 6 Female 0.3165609 February 2018 | ASPIRE Performance Evaluation Report 140 Secondary (n=381) Proportion Balaka Form 1 Male 0.8297598 Form 1 Female 0.6548409 Form 2 Male 0.8289174 Form 2 Female 0.7366236 Machinga Form 1 Male 0.8722639 Form 1 Female 0.6073029 Form 2 Male 0.8713226 Form 2 Female 0.6573105 Zomba Form 1 Male 0.8184721 Form 1 Female 0.7460763 Form 2 Male 0.7038556 Form 2 Female 0.4906911 Table 22a: Sexual practices among primary and secondary students Proportion Std. Err. [95% Conf. Interval] Proportion who have had neither sexual intercourse nor contact Primary (n=580) Male 0.662078 0.043867 0.563323 0.748472 Female 0.810996 0.041228 0.707486 0.883889 Secondary (n=381) Male 0.459765 0.06986 0.312602 0.614295 Female 0.64953 0.071739 0.478713 0.78904 Proportion who report ever having had any type of sexual contact (but not intercourse) Primary (n=580) Male 0.102594 0.018335 0.069577 0.148773 Female 0.10426 0.029212 0.056387 0.184817 Secondary (n=381) Male 0.093345 0.023296 0.052811 0.159745 Female 0.122344 0.023262 0.079211 0.184265 Proportion who report ever having had sexual intercourse Primary* (n=580) Male 0.235328 0.039056 0.162319 0.328306 Female 0.084744 0.024171 0.045486 0.152471 Secondary** (n=381) Male 0.44689 0.071092 0.298576 0.6053 Female 0.228126 0.055248 0.128083 0.372893 * p=0.0005 for difference between sex (significant at 1 percent; design-based F) ** p=0.0023 for difference between sex (significant at 1 percent; design-based F) February 2018 | ASPIRE Performance Evaluation Report 141 Table 22b: Sexual practices among primary and secondary students (disaggregated) Primary (n=580) Proportion Proportion who have had neither sexual intercourse nor contact Balaka Standard 5 Male 0.646863 Standard 5 Female 0.810008 Standard 6 Male 0.497604 Standard 6 Female 0.662632 Machinga Standard 5 Male 0.657021 Standard 5 Female 0.793123 Standard 6 Male 0.524326 Standard 6 Female 0.793429 Zomba Standard 5 Male 0.807711 Standard 5 Female 0.912446 Standard 6 Male 0.874888 Standard 6 Female 0.855729 Proportion who report ever having had any type of sexual contact (but not intercourse) Balaka Standard 5 Male 0.102418 Standard 5 Female 0.12705 Standard 6 Male 0.087093 Standard 6 Female 0.163496 Machinga Standard 5 Male 0.094328 Standard 5 Female 0.169536 Standard 6 Male 0.162572 Standard 6 Female 0.098442 Zomba Standard 5 Male 0.074707 Standard 5 Female 0.03925 Standard 6 Male 0.065831 Standard 6 Female 0.042662 Proportion who report ever having had sexual intercourse Balaka Standard 5 Male 0.250719 Standard 5 Female 0.062942 Standard 6 Male 0.415302 Standard 6 Female 0.173872 Machinga Standard 5 Male 0.248651 Standard 5 Female 0.037341 Standard 6 Male 0.313102 Standard 6 Female 0.108129 Zomba Standard 5 Male 0.117582 Standard 5 Female 0.048304 Standard 6 Male 0.059281 Standard 6 Female 0.101609 Secondary (n=381) Proportion Proportion who have had neither sexual intercourse nor contact Balaka Form 1 Male 0.512435 Form 1 Female 0.685301 Form 2 Male 0.399341 Form 2 Female 0.581035 February 2018 | ASPIRE Performance Evaluation Report 142 Machinga Form 1 Male 0.436076 Form 1 Female 0.746113 Form 2 Male 0.2605 Form 2 Female 0.523288 Zomba Form 1 Male 0.575685 Form 1 Female 0.758383 Form 2 Male 0.444615 Form 2 Female 0.561062 Proportion who report ever having had any type of sexual contact (but not intercourse) Balaka Form 1 Male 0.088383 Form 1 Female 0.105216 Form 2 Male 0.135292 Form 2 Female 0.149316 Machinga Form 1 Male 0.16344 Form 1 Female 0.138952 Form 2 Male 0.14315 Form 2 Female 0.164402 Zomba Form 1 Male 0.075412 Form 1 Female 0.080539 Form 2 Male 0.026682 Form 2 Female 0.138013 Proportion who report ever having had sexual intercourse Balaka Form 1 Male 0.399182 Form 1 Female 0.209483 Form 2 Male 0.465368 Form 2 Female 0.269649 Machinga Form 1 Male 0.400484 Form 1 Female 0.114935 Form 2 Male 0.59635 Form 2 Female 0.31231 Zomba Form 1 Male 0.348903 Form 1 Female 0.161078 Form 2 Male 0.528703 Form 2 Female 0.300926 February 2018 | ASPIRE Performance Evaluation Report 143 ANNEX 4: DOCUMENTS REVIEWED Note: Other information sources are covered in detail in Section 3 and in the References. 1. Ministry of Education, Science and Technology (MoEST) Malawi. Education Management Information System (EMIS) Education Statistics 2014. 2. Ministry of Education, Science and Technology (MoEST) Malawi. Education Management Information System (EMIS) Education Statistics 2015. 3. Ministry of Education, Science and Technology (MoEST) Malawi. 2014. Malawi National Reading Strategy (2014–2019). 4. Psaki, S. R., B.S. Mensch, and E. Solder-Hampejsek. 2017. Associations between Violence in School and at Home and Education Outcomes in Rural Malawi: A Longitudinal Analysis. Comparative Education Review 61(2): 354–390. 5. United States Agency for International Development (USAID). 2017. A Systematic Review of Positive Youth Development Programs in Low- and Middle-Income Countries. 6. United States Agency for International Development (USAID). 2015. ASPIRE Activity Monitoring and Evaluation Plan. 7. United States Agency for International Development (USAID). 2015. ASPIRE Baseline Report: September 2015. 8. United States Agency for International Development (USAID). 2015. ASPIRE Annual Report: Fiscal Year 2015: December 2014–September 2015. 9. United States Agency for International Development (USAID). 2016. ASPIRE Annual Report: Fiscal Year 2016: October 2015–September 2016. 10. United States Agency for International Development (USAID). 2015. ASPIRE FY 2014-15 Annual Work Plan: January–September 2015. 11. United States Agency for International Development (USAID). 2015. ASPIRE FY 2015-16 Annual Work Plan: October 2015–September 2016. 12. United States Agency for International Development (USAID). 2017. ASPIRE Quarterly Report: Quarter 1, Fiscal Year 2017: October 1–December 31, 2016. 13. United States Agency for International Development (USAID). 2017. ASPIRE Quarterly Report: Quarter 2, Fiscal Year 2017: January 1–March 31, 2017. February 2018 | ASPIRE Performance Evaluation Report 144 14. United States Agency for International Development (USAID). 2017. ASPIRE Quarterly Report: Quarter 3, Fiscal Year 2017: April 1–June 30, 2017. 15. United States Agency for International Development (USAID). Malawi ASPIRE Program Overview. Available at: https://www.usaid.gov/malawi/factsheets/malawiaspireprogramoverview 16. United States Agency for International Development (USAID) Malawi. Country Development Cooperation Strategy 2013–2018. Available at: https://www.usaid.gov/malawi/cdcs 17. United States Agency for International Development (USAID) Malawi. Gender Assessment 2012. February 2018 | ASPIRE Performance Evaluation Report 145 ANNEX 5: STATEMENTS OF DIFFERENCE There were no differences of opinion among the evaluation team members. February 2018 | ASPIRE Performance Evaluation Report 146 ANNEX 6: CONFLICT OF INTEREST DISCLOSURES This annex presents conflict of interest disclosure statements for each evaluation team member, as follows:  Nick Shawa, Team Leader  Maureen Chirwa  Zachariah Falconer-Stout  Lynne Franco  Rebecca Frischkorn  Pragati Godbole  Abigail Kazembe  Kelsey Simmons  Sabine Topolansky February 2018 | ASPIRE Performance Evaluation Report 147 DISCLOSURE OF CONFLICTS OF INTEREST Name: Nick Stans Shawa Title: Evaluation Specialist Organization: Encompass LLC Evaluation Position:  Team leader Team member Evaluation Contract or Award Number: other instrument USAID Contract No. AID-OAA-1-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer number(s), if applicable name(s), and award Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15- 00001 I have real or potential conflicts of interest to disclose. Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (I) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: August 28, 2017 February 2018 | ASPIRE Performance Evaluation Report 148 February 2018 | ASPIRE Performance Evaluation Report 149 DISCLOSURE OF CONFLICTS OF INTEREST Name: Maureen Leah Chirwa Title: Evaluation Specialist Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 25th August, 2017 February 2018 | ASPIRE Performance Evaluation Report 150 DISCLOSURE OF CONFLICTS OF INTEREST Name: Zachariah Falconer-Stout Title: Evaluation Manager Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: August 24, 2017 February 2018 | ASPIRE Performance Evaluation Report 151 DISCLOSURE OF CONFLICTS OF INTEREST Name: Lynne Franco Title: Vice President, Technical Assistance & Evaluation Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: August 25, 2017 February 2018 | ASPIRE Performance Evaluation Report 152 DISCLOSURE OF CONFLICTS OF INTEREST Name: Rebecca Frischkorn Title: Evaluation Specialist Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 25/08/2017 February 2018 | ASPIRE Performance Evaluation Report 153 DISCLOSURE OF CONFLICTS OF INTEREST Name: Pragati Godbole Title: Evaluation Specialist Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: Aug 29th, 2017 February 2018 | ASPIRE Performance Evaluation Report 154 February 2018 | ASPIRE Performance Evaluation Report 155 DISCLOSURE OF CONFLICTS OF INTEREST Name: Kelsey Simmons Title: Evaluation Specialist Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable Girls Empowerment through Education and Health (ASPIRE) activity in Malawi, implemented by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: August 29, 2017 February 2018 | ASPIRE Performance Evaluation Report 156 DISCLOSURE OF CONFLICTS OF INTEREST Name: Sabine K. Topolansky Title: Project Coordinator Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: USAID Contract No. AID-OAA-I-15-00021 Contract or other instrument Task Order No. AID-612-TO-17-00002 USAID Project(s) Evaluated: Girls Empowerment through Education and Include project name(s), implementer name(s), and award Health (ASPIRE) activity in Malawi, implemented number(s), if applicable by Save the Children under Cooperative Agreement No. AID-612-A-15-00001 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not If yes, I disclose the following facts: limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: August 30, 2017 February 2018 | ASPIRE Performance Evaluation Report 157 ANNEX 7: BACKGROUND ON EVALUATION TEAM MEMBERS NICHOLAS (NICK) SHAWA, MA Position: Team leader Mr. Shawa has over 20 years of experience conducting evaluation and research in Southern Africa and especially his home country, Malawi. He has worked extensively on USAID-funded education programs and is deeply familiar with the USAID/Malawi education results framework, indicators, and context. Mr. Shawa has successfully led a number of mixed-methods evaluations of education assistance programs in Malawi, including design and implementation of data collection instruments, data analysis, and production of high-quality reports. Mr. Shawa also has extensive experience managing learner achievement studies. His monitoring and evaluation (M&E) systems were highly praised during the Data Quality Assurance audit of the USAID-funded Malawi Teacher Professional Development Support (MTPDS). As the Senior Monitoring and Evaluation Systems Advisor for this activity, in 2011, he supported MoEST in the design and development of the M&E framework and strategy for the National Primary Curriculum under Primary Curriculum and Assessment Reform (PCAR). As part of this work, he built M&E capacity in the Directorate of Inspection and Advisory Support and Directorate of Teacher Development. Additionally, he supported the design and development of the EGRA in Malawi in 2010. On Improving Education Quality II Project, Mr. Shawa supported literacy team experts in developing EGRA tools and conducted data collection to manage a multiyear longitudinal data on student performance based at the Malawi Institute of Education. Mr. Shawa has a Master’s degree in Development Studies and is currently pursuing a PhD in Evaluation Studies. ZACHARIAH FALCONER-STOUT, MA Position: Project Manager Mr. Falconer-Stout has over 8 years’ experience implementing and evaluating youth and education programs, specializing in quantitative analysis, managing large data collection operations, and designing mixed-methods studies. He has worked domestically and internationally in Central America, Eastern Europe, Southeast Asia, and Sub-Saharan Africa. Mr. Falconer-Stout has been lead author on 15 evaluation reports and technical resources, providing deliverables for clients including February 2018 | ASPIRE Performance Evaluation Report 158 USAID, Bill & Melinda Gates Foundation, MacArthur Foundation, and the U.S. Peace Corps. His education work has focused on early grade literacy, youth life skills programs, and parental and community engagement in school management. He has developed several classroom observation tools for both formative and summative assessment, and is experienced in integrating gender considerations in studies in diverse country contexts. He has additional research experience in democratization, governance, and human rights issues. Mr. Falconer-Stout’s recent portfolio includes managing the 5-year research and evaluation program for USAID’s Time to Learn project in Zambia, and leading two EGRA surveys with a combined sample of over 6,000, four performance evaluations, and a 6-study series of case studies. Additional experience with EnCompass includes leading evaluations of a community-based early grade reading program in Bangladesh, a digital literacy program in Myanmar, and contributing to the USAID/Honduras Gender-Based Violence Assessment and USAID/Lebanon Basic Education Gender Analysis. Combined with other projects, Mr. Falconer-Stout has performed almost 20 weeks in the field collecting data, conducted and coded over 210 qualitative interviews and focus groups, and has provided 25 weeks of short-term technical assistance. Mr. Falconer-Stout holds an MA with highest honors in International Political Economy from Central European University in Budapest, and is also a Senior Fellow with Humanity in Action. He is well versed in Stata 14 and qualitative data analysis software. MAUREEN LEAH CHIRWA, PHD Position: Evaluation Specialist Dr. Chirwa has extensive experience conducting evaluations in the health sector in Malawi, with a focus on gender integration in studies. She has led numerous studies focusing on gender in health and education projects. She specializes in qualitative data collection, especially data on sensitive subjects (e.g., gender-based violence and HIV), and is well versed in interviewing children to obtain sensitive qualitative data. Dr. Chirwa’s relevant experiences include leading an external evaluation for the Mary’s Meals Malawi and Zambia, and a baseline survey of Chigodi Area M-LEARN and M-WASH projects for World Vision Malawi, which included an assessment of learners’ reading skills. For ICCO, she conducted mapping of sexual and reproductive health services for youth aged 10 to 18 in Lilongwe and Mzimba South Districts. Additionally, for Girl Effects – London, she conducted a qualitative formative research Girls’ Voice Counts, engaging 10- to 19-year olds to listen and learn from them on their perspective of girls in Malawi. For the UN- and EU-funded project supporting the Ministry of Gender, Children and Social Welfare in Malawi, Prof. Chriwa conducted the National Sectoral Gender Needs Assessment of the extent to which National Sectoral policies have integrated gender February 2018 | ASPIRE Performance Evaluation Report 159 into their policies, strategies, and procedures for the Gender Equality and Women Empowerment Program. Prof. Chirwa holds a PhD in Gender Management and MA in Management and Research. LYNNE FRANCO, SCD Position: Corporate Monitor Dr. Franco brings 30 years of experience leading complex, multi-stakeholder evaluations, as well as in the design, implementation, and management of implementation research, and in the use of facilitation and participatory technical assistance and evaluation techniques. Dr. Franco has designed evaluations that have sought to address capacity building, research, scale, institutionalization, and country ownership. She implemented methods including Appreciative Inquiry, organizational network analysis, online surveys, in-depth interviews, creative group data collection, and household surveys, which are particularly effective in such evaluations. Dr. Franco currently serves as the team leader on long-term evaluation partner contracts for the MacArthur Foundation’s Big Bet On Nigeria portfolio and Save the Children’s Saving Newborn Lives Initiative. She has also led shorter-term evaluations, such as the evaluation of USAID’s Bureau of Policy, Planning and Learning’s Program Cycle, PEPFAR Caribbean Regional Program, and the Bill & Melinda Gates Foundation’s Maternal Health Task Force and African Tobacco Control Consortium. Over her career, Dr. Franco has held long-term positions in Benin, Malawi, and Mali, and worked throughout Africa, Eastern Europe, Latin America, and the Middle East. She has authored peer review publications on topics such as quality improvement, health policy reform, programming for orphans and vulnerable children, social participation in and impact of community-based health insurance, and network analysis of organizations working on stillbirth issues. Dr. Franco holds an ScD in International Health Systems and an MHS in Health Planning from Johns Hopkins University School of Hygiene and Public Health. REBECCA FRISCHKORN, PHD Position: Qualitative Data Analyst Based in Nairobi, Kenya, Dr. Frischkorn has lived and worked extensively in Southern Africa, researching refugee and migration issues, facilitating conflict mitigation trainings with NGOs, and evaluating USAID’s basic education activities. Her M&E experience has focused on global environmental projects, and includes analysis of impact and performance data, project strategies and outcomes, and M&E tools and methods. Dr. Frischkorn recently joined Samuel Hall as a Project Manager in Nairobi, where she works on migration-related research projects in the East African Community and the Horn of Africa. Previously, she worked as a Research and Evaluation Specialist February 2018 | ASPIRE Performance Evaluation Report 160 at EnCompass LLC for USAID’s Time to Learn project in Zambia, which supported improved early grade literacy in community schools, and educational delivery for orphans and vulnerable children. Dr. Frischkorn received a Fulbright to complete her dissertation research on urban refugees in Lusaka, Zambia and holds a PhD in Cultural/Social Anthropology from American University in Washington, D.C. PRAGATI GODBOLE, PHD Position: Quantitative Data Analyst Dr. Godbole has extensive experience in quantitative and qualitative research and analysis, with a focus on girls’ education and international education policy. Dr. Godbole recently served as a Senior Statistician for USAID’s Early Grade Reading project in Afghanistan and as a Data Analyst for EnCompass’ endline evaluation of the USAID Time to Learn activity in Zambia. For these projects, her responsibilities included developing surveys, monitoring data from the field, cleaning and analyzing data, drafting report sections on methodology, results, and findings, as well as producing comprehensive summary tables. In 2016, Dr. Godbole served as an adjunct lecturer at the American University where she taught two courses: Quantitative Research in Education and Education for International Development. In 2015, Dr. Godbole served as an Indicator Specialist with Moira Wilkinson Consulting where she re￾conceptualized education indicators for concept note on post-2015 global education goals, created measurable post-2015 education indicators, and developed companion guidelines for implementing framework. Dr. Godbole holds a PhD in Teaching, Learning, Policy and Leadership from the University of Maryland. ABIGAIL KAZEMBE, PHD Position: Evaluation Specialist Dr. Kazembe is an experienced evaluator and researcher with extensive experience in the Malawian health sector. She has conducted numerous research studies pertaining to sexual and reproductive health, including among school-age girls. Over her 13-year career, Dr. Kazembe has worked on evaluating development programs funded by organizations such as USAID, WHO, UNICEF, UNFPA, Save the Children, World Vision, and the World Bank, among others. She has strong analytical skills and is well versed in a number of data collection and analysis approaches. In 2014, Dr. Kazembe served as the co-investigator for the Evaluation of Youth-Friendly Health Services funded by USAID, UNICEF, WHO, and UNFPA. In addition, from 2008 to 2013, she worked as a researcher for Hope for the Future, a school-based HIV prevention program for youth in Malawi. In 2005 and 2006, she also conducted a research study on female adolescent sexual February 2018 | ASPIRE Performance Evaluation Report 161 decision making in the context of HIV and AIDS among high school students in Malawi. Dr. Kazembe holds a PhD in Public Health Sciences and an MSc in Nursing. KELSEY SIMMONS, MA Position: Evaluation Specialist Ms. Simmons supports EnCompass’ technical assistance and evaluation work, contributing to resource development, as well as design and implementation of qualitative and quantitative evaluations of international and domestic projects. She has 7 years of experience living and working throughout Sub-Saharan Africa, researching, designing, and evaluating programs in leadership development, gender-based violence, HIV/AIDS, maternal and child health, malaria, and health systems strengthening. Ms. Simmons’ most recent portfolio of work includes leading the endline evaluation of Save the Children’s Saving Newborn Lives grant funded by the Bill & Melinda Gates Foundation, as well as leading the Nigeria Case Study for the USAID-supported MEASURE Evaluation mid-performance evaluation. Previously, Ms. Simmons managed EnCompass’ evaluation work for the MacArthur Foundation’s portfolio of grants on maternal health accountability in Nigeria. She also led the qualitative analysis of an Organizational Network Analysis of the community of organizations working on stillbirths, supported the evaluation of UN Women’s regional architecture, and led the case study of the PEPFAR-supported Gender-Based Violence Initiative in Tanzania, which identified effective practices for integrating gender-based violence programming into the existing HIV/AIDS programs. Ms. Simmons has presented EnCompass’ work at international conferences, including the American Evaluation Association Conference (2016), African Evaluation Association Conference (2017), and USAID Global Health Mini-University (2017). Prior to joining EnCompass, Ms. Simmons worked for USAID's Global Health Bureau in the Office of HIV/AIDS. She also served as a Community Health Volunteer with the Peace Corps in Zambia. She holds an MA in International Development Studies with concentration in Global Health Monitoring and Evaluation from the Elliott School of International Affairs at the George Washington University. She is fluent in English, and has limited working proficiency in French and iCibemba. SABINE K. TOPOLANSKY, MA Position: Evaluation Associate Ms. Topolansky is an international affairs professional with over 4 years of experience, including an internship at UNESCO headquarters in Paris, France. In her current role of Evaluation Associate for the Technical Assistance and Evaluation team at EnCompass, Ms. Topolansky has supported February 2018 | ASPIRE Performance Evaluation Report 162 analysis of qualitative and quantitative data using a variety of software programs, conducted document reviews and interviews, contributed to development of data collection instruments, administered surveys, and helped draft evaluation designs. Her most recent assignments include support to the midline and endline evaluations of MacArthur Foundation’s maternal health accountability portfolio in Nigeria, and the EnCompass-led evaluation of the U.S. State Department’s programming focused on lesbian, gay, bisexual, transgender, and intersex issues in Central America, Eastern Europe, and Sub-Saharan Africa. Ms. Topolansky also has experience working on bioethics, gender issues, HIV, public health, and gender-based violence, and has recently co-authored a report related to gender equality. Ms. Topolansky holds an MA in International Affairs from the George Washington University. February 2018 | ASPIRE Performance Evaluation Report 163 ANNEX 8: DATA COLLECTION TOOLS The complete set of data collection tools are provided under separate cover. Please refer to the supplemental document, “Performance Evaluation of Girls Empowerment through Education and Health (ASPIRE) in Malawi: Annex 8.” United States Agency for International Development 1300 Pennsylvania Avenue, NW Washington, D.C. 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov