SINDH COMMUNITY MOBILIZATION PROGRAM MIDTERM PERFORMANCE EVALUATION DECEMBER 28, 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International, A Tetra Tech Company. SINDH COMMUNITY MOBILIZATION PROGRAM MIDTERM PERFORMANCE EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Sindh Community Mobilization Program – Midterm Performance Evaluation iii ACKNOWLEDGMENTS This report represents a joint effort of a large number of people. Jared Berenter was the primary author. Management Systems International (MSI) staff in Pakistan planned and managed the evaluation, developed the data collection instruments, led the data analysis, and contributed substantially to the writing. Monitoring and Evaluation Specialist Saeed Ahmad served as field team lead. Dr. Hina Hussain Kazmi, Rehana Paveen Shaikh, Shahzad Ahmed, Waseem Abbas, and Khurshid Khan provided field support, expert advice, and local context in designing and implementing the evaluation and interpreting the data. Babur Wasim, MSI’s statistics and evaluation specialist, contributed to developing the evaluation sample and to analyzing the data. Finally, the respondents who gave generously of their time and knowledge to participate in surveys and interviews deserve special thanks. Sindh Community Mobilization Program – Midterm Performance Evaluation iv CONTENTS Acronyms...........................................................................................................................vi Definitions .........................................................................................................................vii Program Summary ...........................................................................................................ix Executive Summary.......................................................................................................... 1 Evaluation Purpose and Questions.................................................................................. 4 Theory of Change................................................................................................................................ 4 Evaluation Questions.......................................................................................................................... 4 Program Background........................................................................................................ 4 Sindh Education Sector Reform ....................................................................................................... 5 Sindh Basic Education Program........................................................................................................ 5 Community Mobilization Program .................................................................................................. 6 Evaluation Methods and Limitations............................................................................... 6 Data Sources......................................................................................................................................... 6 Sampling for School-Level Primary Data Collection ................................................................... 7 Data Analysis Methods....................................................................................................................... 8 Data Limitations................................................................................................................................... 8 Findings ............................................................................................................................ 10 Question 1: CMP’s Contribution to Access to Schooling .......................................................10 Question 2: Effectiveness of CMP Community Engagement Activities................................18 Question 3: CMP Support for Education Sector Reforms......................................................24 Question 4: Relevance of CMP Performance Indicators and Targets..................................29 Conclusions ...................................................................................................................... 32 Recommendations .......................................................................................................... 34 Annexes............................................................................................................................ 36 Annex 1: Statement of Work .........................................................................................................36 Annex 2: Assignment Work Plan...................................................................................................42 Annex 3: Team Composition and Conflict of Interest Disclosures......................................65 Annex 4: Interviews Conducted by the Evaluation Team........................................................66 Annex 5: Data Collection Instruments.........................................................................................67 Annex 6: Documents Reviewed...................................................................................................105 Annex 7: Sampling ...........................................................................................................................107 Annex 8: Background on Sindh Education Sector and Sector Reforms.............................111 Annex 9: Impact of CMP on Enrollment....................................................................................113 Annex 10: SMC Subcommittee Membership, by Sex..............................................................120 Sindh Community Mobilization Program – Midterm Performance Evaluation v List of Tables Table 1: Program Summary............................................................................................................................. ix Table 2: Key Informant Selection..................................................................................................................66 Table 3: Population Frame............................................................................................................................107 Table 4: Sample Frame ..................................................................................................................................108 Table 5: Group Interview Sample...............................................................................................................108 Table 6: Distribution of Sampled Schools.................................................................................................109 Table 7: Sampled Schools by Location.......................................................................................................110 Table 8: ITSA Regression Output...............................................................................................................115 Table 9: Girls’ Enrollment in Primary Schools.........................................................................................117 Table 10: Boys’ Enrollment in Primary Schools.......................................................................................117 Table 11: Girls’ Enrollment in Higher than Primary Schools...............................................................117 Table 12: Boys’ Enrollment in Higher than Primary Schools................................................................118 Table 13: Girls’ Total Enrollment in all Levels.........................................................................................118 Table 14: Boys’ Total Enrollment in all Levels.........................................................................................118 Table 15: Total Enrollment, Boys and Girls at all Levels.......................................................................119 Table 16: SMC Subcommittee Membership, by Sex...............................................................................120 List of Figures Figure 1: Map of Program Area ....................................................................................................................... x Figure 2: Top Factors Affecting Parents’ Enrollment Decisions............................................................11 Figure 3: Comparison of Factors Affecting Parents’ Enrollment and Retention Decisions......................................................................................................................................14 Figure 4: Summary of CMP’s Contribution to Enrollment.....................................................................17 Figure 5: Interpretation of Coefficients.....................................................................................................115 Figure 6: Summary of CMP Contribution to Enrollment......................................................................116 Sindh Community Mobilization Program – Midterm Performance Evaluation vi ACRONYMS CFC Charter for Compassion CMP Community Mobilization Program DEO District Education Officer DST District Support Team EMO Education Management Organization GoS Government of Sindh IBA Institute of Business Administration IRD International Relief and Development MSI Management Systems International PPP Public-Private Partnership PSLM Pakistan Social and Living Standards Measurement Survey RFP Request for Proposals RSU Reform Support Unit (GoS) SBEP Sindh Basic Education Program SELD School Education and Literacy Department (GoS) SEMIS Sindh Education Management Information System SMC School Management Committee TEO Taluka Education Officer USAID United States Agency for International Development USG United States Government Sindh Community Mobilization Program – Midterm Performance Evaluation vii DEFINITIONS The evaluation team developed the following definitions and measurements: • Attendance: A student’s physical presence in the classroom. • Campus schools: Schools into which other schools have been merged or consolidated under the Sindh education reforms. Campus schools serve as the administrative and/or physical hub for the other merged schools. • Consolidation: Distinct government schools operating on the same campus, or catering to the same child population in a local community, converted into a single functional school.1 • Construction schools: Schools newly constructed under the Sindh Basic Education Program. • Dropout: Students are considered dropped out after 90 consecutive days of absence from school. • Education management organization (EMO): Private sector organization tasked by the Government of Sindh School Education and Literacy Department to manage government-owned schools. EMOs are selected through a competitive bidding process. Under the first request for proposals from EMOs issued by the Government of Sindh, school management contracts were issued to Sukkur Institute of Business Administration and Charter for Compassion. Other EMOs that have won school management contracts include the Indus Resource Centre and Beaconhouse. • Education manager: Divisional directors of education, district education officers, and taluka education officers. • Enrollment: All students on the general register of a school. For purposes of this evaluation, terms relating to enrollment (including new enrollment and re-enrollment) are defined in the Community Mobilization Program’s Monitoring, Evaluation, and Learning Plan under Indicator 1.B, the second of two goal-level indicators pertaining to enrollment (referred to as “Goal-b”). • Gross enrollment rate: The Pakistan Social and Living Standards Measurement defines the gross enrollment rate as the number of children attending primary, middle, or matric school, divided by the number of children of respective age for that level. • Neighboring schools: Schools that received CMP community engagement and mobilization interventions and are eligible for small grants for school improvement but are not newly constructed. • New enrollment: Students of school age who never previously enrolled to study in a formal education setting but have become enrolled through United States Government assistance. • Re-enrollment: Students who were previously enrolled in a formal education setting but discontinued study and have since become re-enrolled. 1 School consolidation covers three types of schools: (a) adjoining schools consolidated administratively; (b) embedded schools or schools sharing the same premises; (c) schools that are within a stipulated distance of each other, with one strategic school among them having the horizontal and/or vertical space for expansion to accommodate students from merged schools. See Government of Sindh, School Education and Literacy Department, http://www.sindheducation.gov.pk/educationReforms/sc.jsp. Sindh Community Mobilization Program – Midterm Performance Evaluation viii • Retention: Graduation from one grade and enrollment in the next grade. (Pakistan’s Constitution provides for 10 years of free and compulsory education.) • School management committee (SMC): The SMC is a participatory body established and notified in every public school to support school management, facilitate school improvements, ensure financial accountability, and monitor education activities. The SMC is comprised of an executive body and a general body. The executive body includes a secretary (school head teacher), elected chairperson (typically a parent or influential member of the community), and three elected members (also typically parents). The broader SMC general body includes all parents or guardians of enrolled students. • Subcommittee: The Community Mobilization Program has formed SMC subcommittees at the school level to increase community involvement and engage more parents and community members in school management activities. The program initially formed five subcommittees which were later restructured into three, namely (a) social mobilization and school improvement planning committee, (b) local resource and grants management committee, and (c) school health and nutrition committee. Subcommittees generally are comprised of one representative of the SMC executive body and volunteers from the SMC general body. • Upgradation: Upgradation refers to the process for formally upgrading existing primary level schools to middle-level schools, and middle/elementary-level schools to high schools, to reduce the enrollment gap between primary and secondary education. Upgradation is often accompanied by provision of additional teachers and facilities. Sindh Community Mobilization Program – Midterm Performance Evaluation ix PROGRAM SUMMARY TABLE 1: PROGRAM SUMMARY Title/Field Program Information Start date August 7, 2013 Location Sindh Province (Dadu, Jacobabad, Karachi, Kashmore, Khairpur, Larkana, Qambar Shahdadkot, and Sukkur districts) Implementing partner(s) International Relief and Development (IRD) Novated to Blumont Engineering Solutions Inc. on December 29, 20172 USAID/Pakistan Mission Strategic Framework objectives addressed USAID/Pakistan’s Education Results Framework Development Objective 4: Improved opportunities for learning and work • IR 4.1: Improved educational access • IR 4.3: Improved accountability in education 2 The evaluation took place before Blumont Engineering Solutions Inc. began implementing CMP in December 2017. This report mentions IRD as the implementing partner since the findings refer to activities conducted under IRD. When relevant, the report directs recommendations to Blumont Engineering Solutions Inc. as the party responsible for implementing recommendations. Sindh Community Mobilization Program – Midterm Performance Evaluation x FIGURE 1: MAP OF PROGRAM AREA Sindh Community Mobilization Program – Midterm Performance Evaluation 1 EXECUTIVE SUMMARY Evaluation Purpose and Questions The purpose of the Community Mobilization Program (CMP) midterm evaluation is to assess CMP’s progress toward achieving program objectives. The evaluation includes a review of key performance indicators and targets and recommends strategies to improve the measurement of program performance. Evaluation findings will inform future USAID programming decisions for community engagement and mobilization in the education sector. The evaluation addresses the following questions: 1. How and to what extent has CMP contributed to improving access to schooling for boys and girls in target communities? 2. To what extent have CMP processes been effective for engaging target communities? 3. To what extent have CMP’s strategies in support of Government of Sindh (GoS) education reforms, specifically school consolidation and public-private partnerships, been effective for achieving GoS reform objectives? 4. Are CMP indicators appropriately defined, to what extent has CMP achieved its targets, and are these targets relevant to achieving program objectives? Summary of Findings and Conclusions CMP provides a valuable contribution to the Sindh Basic Education Program (SBEP), facilitating SBEP’s infrastructure components through community engagement at the school level and advocacy and technical assistance at the policy level. The evaluation finds that CMP’s development hypothesis holds true, namely that activities that build community awareness of education processes and engage parents in those processes have a positive effect on access to schooling for boys and girls. CMP interventions contend with a multiplicity of factors—internal to both schools and households— that affect parents’ enrollment decisions. The perceived costs and benefits of enrolling one’s child in school vary for boys and girls and evolve as children grow. Physical and affordable access to schooling, with adequate facilities and quality instruction, are key factors driving enrollment decisions. Within this decision-making context (and to the extent that enrollment serves as a meaningful indicator of access), growth in enrollment during CMP’s period of performance demonstrates that CMP engagement strategies have increased access to schooling. Feedback from parents, head teachers, and school management committees (SMCs) suggests that CMP has contributed to the socialization of education— in other words, to a collective recognition of the accessibility and desirability of education. The SMC is the chief mechanism by which CMP has engaged with target communities. The evaluation found that SMCs are appropriate institutional vehicles for community engagement aimed at increasing enrollment. SMCs are legally notified and participatory in their composition, have authority and responsibility to spend government funds on school improvements, and enjoy a sense of ownership from government institutions in the education sector. The evidence suggests that CMP’s strategies to activate and strengthen SMCs have increased SMCs’ capacity to engage communities and provide support for school management, and thus contributed to enrollment. CMP’s capacity-building efforts Sindh Community Mobilization Program – Midterm Performance Evaluation 2 have provided SMCs and parents with clarity on SMC roles and responsibilities, strengthened SMC processes for financial management and planning, and instilled SMC executive and general bodies with knowledge and confidence to fulfill their intended purposes. With CMP’s assistance, newly formed SMC subcommittees have further broadened SMCs’ scope and community involvement in SMC activities. CMP’s efforts to engage and strengthen SMCs have increased interaction between schools and communities, as well as parents’ active involvement in school affairs. CMP strategies have also facilitated interaction between schools and education officials, creating opportunities to mobilize government resources for school improvement and upgradation. In addition to community mobilization and strengthening of school management processes, CMP supports two Sindh education sector reforms: (1) private sector management of government schools under the Public-Private Partnership (PPP) Act of 2010 and (2) the School Consolidation Policy of 2012. CMP has supported these reforms at both the provincial and school levels with mixed results. With respect to education reforms under the PPP Act, CMP’s provision of resources and technical assistance to the PPP Node at the Sindh School Education and Literacy Department (SELD) has been integral to establishing standard operating procedures, successfully piloting the education management organization (EMO) model, and engaging PPP stakeholders at the provincial and district levels. Although post-EMO transition support for school management lies outside CMP’s scope, CMP actively supports EMOs in the conduct of community engagement activities. However, EMO respondents expressed a lack of clarity regarding the roles and responsibilities of CMP, EMOs, and government stakeholders in the implementation of the EMO model, which suggests a need for continued institutional coordination in the post-transition phase to achieve GoS reform objectives. With respect to the School Consolidation Policy, the GoS’s Reform Support Unit (RSU)3 reported little need for regular CMP technical inputs toward consolidation reforms at the provincial level. At the district and school levels, CMP has eased implementation of consolidation by providing education managers and head teachers with orientation around consolidation processes and procedures. Sensitization and technical assistance for consolidation at the school level, however, fall within the purview of CMP’s conventional activities. District education officers (DEOs), for example, reported CMP contributions to consolidated schools in the form of small grants and identification of needs for school improvement. Finally, the evaluation assesses the suitability of enrollment as an indicator of access to schooling. The evaluation finds enrollment to be an insufficient, and potentially misleading, indicator of access for two reasons. First, enrollment does not capture attendance. External factors such as economic need, household obligations, or lack of transportation curtail attendance even if they do not curtail enrollment. Second, most sampled CMP schools were fully enrolled and did not have capacity for additional enrollment. In such cases, enrollment does not effectively capture CMP’s performance. Likewise, aggregate enrollment targets established in the CMP Monitoring, Evaluation, and Learning Plan do not distinguish enrollment attributable to CMP, account for transfers of students from one school to another, or account for the longer-term impact of CMP’s community engagement efforts. Recommendations Based on these findings, the evaluation team recommends the following: 1. To better capture the effectiveness of education programs for improving access to schooling, USAID should include attendance and retention as indicators for access to schooling in future 3 The RSU is the chief entity responsible for policy formation and oversight of consolidation reforms. Sindh Community Mobilization Program – Midterm Performance Evaluation 3 programming. USAID may also consider monitoring school capacity and out-of-school children at the community level over time to better understand changes in new enrollment, dropout rates, and retention, and inform realistic enrollment targets based on the local context. 2. CMP should have a strategic plan to mobilize or advocate for stopgap supply-side interventions to address overenrollment or accommodate students displaced by new school construction. Where appropriate and practical, CMP should shift to attendance and retention activities in overcrowded schools. 3. Activities to strengthen SMC processes for engagement and school improvement should include an explicit knowledge-building component for SMC general bodies concerning roles, responsibilities, and practices of the SMC. Forums for such activities include SMC general body meetings and SMC subcommittee meetings. 4. CMP should establish formal mechanisms to transfer knowledge and best practices for EMO management to the PPP Node and district support teams (DSTs). 5. CMP should implement a plan to institutionalize stakeholder coordination at the district level to support EMOs before and after the transition to EMO school management. This action may require modification to CMP’s current scope. 6. CMP should conduct a needs assessment exercise based on lessons learned from consolidated schools to assist RSU in identifying implementation challenges and improving implementation of the School Consolidation Policy. Sindh Community Mobilization Program – Midterm Performance Evaluation 4 EVALUATION PURPOSE AND QUESTIONS The CMP midterm evaluation assesses CMP’s progress toward achieving program objectives, namely to improve access to schooling, particularly for girls, and support GoS education sector reforms. The evaluation identifies lessons learned about whether, how, and under what conditions community engagement has been effective for improving access to schooling. The evaluation includes a review of select CMP performance indicators and targets and recommends strategies to improve the suitability of these indicators and targets for measuring program performance. Evaluation findings will inform future USAID programming decisions with respect to how community engagement can best support SBEP and other education sector programs in Sindh. Theory of Change The CMP theory of change posits that community engagement and mobilization in the education process build community awareness of education processes, enhance coordination between communities and district governments to expand girls’ enrollment, and increase community involvement in school management and GoS education sector reforms, thereby improving access to schools for boys and girls. In the context of CMP, community engagement and mobilization include activities such as enrollment drives, community events, and other outreach efforts aimed at increasing enrollment and parental participation in school affairs; development and implementation of social mobilization and school improvement plans with communities and school stakeholders; strengthening of SMCs and formation of participatory subcommittees in target schools; and capacity building and engagement of DEOs. Education sector reforms include merging, consolidating, and upgrading schools and piloting a PPP model for school management that outsources school management to EMOs. Evaluation Questions Based on the theory of change, the evaluation addresses the following evaluation questions: 1. How and to what extent has CMP contributed to improving access to schooling for boys and girls in target communities? 2. To what extent have CMP processes been effective for engaging target communities? 3. To what extent have CMP’s strategies in support of GoS education reforms, specifically school consolidation and public-private partnerships, been effective for achieving GoS reform objectives? 4. Are CMP indicators appropriately defined, to what extent has CMP achieved its targets, and are these targets relevant to achieving program objectives? PROGRAM BACKGROUND The following details education sector reforms in Sindh and the respective roles of SBEP and CMP in supporting these reforms. Sindh Community Mobilization Program – Midterm Performance Evaluation 5 Sindh Education Sector Reform Despite constitutionally guaranteed rights to free and compulsory education in Pakistan, the GoS faces significant challenges ensuring universal access to schooling. Gross enrollment rates in Sindh have stagnated or declined in recent years—a trend exacerbated by damages to government schools inflicted by floods in 2010—and lag significantly behind both Punjab and Khyber Pakhtunkhwa.4 Gross enrollment remains significantly lower for girls than for boys and declines precipitously after primary school. The GoS has undertaken measures to address these trends and other challenges in the education sector. At the policy level, the GoS has implemented two medium-term strategic frameworks for sector reform: the Sindh Education Reform Plan, implemented from 2006–2012, and the Sindh Education Sector Plan, implemented from 2014–2018. This evaluation considers two specific reforms to improve school management and accountability: • School consolidation: In 2012, the GoS introduced the School Consolidation Policy to make better use of limited resources, eliminate unviable schools, and increase access to higher-level schooling. School consolidation placed adjoining schools, embedded schools (i.e., schools situated on the same premises), or schools within a stipulated distance of each other under single management structures. Under the School Consolidation Policy, one school may provide physical space to accommodate students from merged schools, or a central “campus” school may oversee the administrative functions of neighboring schools. The consolidation process is managed by the RSU and implemented by the regional directors of education in Sindh. • Public-private partnerships for school management: Private sector management of public sector schools is a long-standing practice in Sindh. Such arrangements have historically been loosely structured and often philanthropic in nature. The PPP Act, passed in 2010, introduced a structured framework for engaging the private sector in public sector service provision. In the education sector, the act provided legal cover for outsourcing school management and education service delivery to the private sector. To this end, SELD established a PPP Node tasked with developing a regulatory framework for partnering with the private sector to provide education services in public schools. With CMP’s assistance, the PPP Node is piloting a model for outsourcing public sector school management to private sector entities known as EMOs (see Annex 8). Sindh Basic Education Program In 2012, in collaboration with the GoS, USAID launched SBEP to support the GoS Sindh Education Reform Plan, with a focus on increasing and sustaining enrollment at the primary, middle, and secondary levels in seven districts of northern Sindh and five towns of Karachi. SBEP includes six components: (1) construction of schools affected by the 2010 floods in the program districts (106 schools constructed under SBEP received CMP support and will transition to EMO management); (2) support for GOS policy reforms to merge, consolidate, and upgrade schools through construction of schools; (3) improvement in early grade reading in primary schools; (4) community mobilization, with a focus on increasing girls’ enrollment and improving the nutritional status of children; (5) technical assistance to SELD; and (6) architect and engineering support for school construction. 4 Without estimating the overall effect of the 2010 floods on gross enrollment, a preliminary damage and needs assessment conducted by the World Bank, Asian Development Bank, and Government of Pakistan estimated that 5,655 schools in Sindh were partially damaged or completely destroyed by the floods—more than in any other province. This constitutes approximately 12 percent of the 47,552 schools for which SEMIS data was available for the 2010–2011 school year. Sindh Community Mobilization Program – Midterm Performance Evaluation 6 Community Mobilization Program As a cornerstone of SBEP, CMP supports the other SBEP components as well as GoS education reform efforts. CMP aims to sustain community mobilization and school-based management by focusing on four key objectives: 1. Increasing communities’ involvement in GoS education reform of merging, consolidating, and upgrading schools. 2. Improving coordination between the community and district administration for increased girls’ enrollment. 3. Improving child nutrition in selected communities and government schools through research that informs innovation and good practice. 4. Sustaining community mobilization and school-based management through PPPs. Under a newly conceived PPP model, the GoS has initiated a process of outsourcing the management of public sector schools to reputable EMOs. CMP is implemented by International Relief and Development (IRD)5 in partnership with Health and Nutrition Development Society, Indus Resource Center, and Agha Khan University – Human Development Program. CMP is implemented in Sukkur, Khairpur, Larkana, Qambar Shahdadkot, Jacobabad, Kashmore, and Dadu districts in northern Sindh, and in Kemari, Lyari, Orangi, Gadap, and Bin Qasim towns of Karachi.6 EVALUATION METHODS AND LIMITATIONS The evaluation team employed a mixed methods approach, relying on both quantitative and qualitative data from multiple sources to triangulate findings and generate credible and robust evidence to answer the evaluation questions. The following sections detail data sources, sampling, and data limitations.7 Data Sources Quantitative, qualitative, and other secondary data sources included the following: Quantitative Data Secondary data from the SEMIS school census provided by the RSU constituted the evaluation’s principal source of quantitative data. The evaluation team received full or partial SEMIS data for 29 districts and approximately 49,700 schools in Sindh for academic years 2008–2009 through 2015–2016. School-level indicators contained in the SEMIS data included boys’ and girls’ enrollment, infrastructure (e.g., number of rooms and number of classrooms), facilities (e.g., boundary walls, water, washrooms, and electricity), and number of teachers, among others. SEMIS served as the primary source of data for generating a comparison group of schools to estimate CMP’s effect on enrollment under Question 1. 5 On December 29, 2017, USAID novated the project from IRD to Blumont Engineering Solutions Inc. 6 For additional background information and resources on the status of the education sector in Sindh, development of school management committees, and recent education sector reforms in Sindh, see Annex 8. 7 For a more detailed explanation of how data sources and analysis methods were used to address each evaluation question, see Table 2, “Summary of Proposed Data Collection and Analysis Methods,” in the assignment work plan in Annex 2. Sindh Community Mobilization Program – Midterm Performance Evaluation 7 The evaluation drew additional quantitative data from CMP’s performance monitoring reporting. Specifically, the evaluation team used program monitoring data to assess the extent to which CMP met its enrollment targets and the appropriateness of those targets under Question 4. Qualitative Data The evaluation team collected qualitative data through group and key informant interviews at the school, district, and provincial levels (see Annex 4 for a full inventory of interviews conducted). At the school level, the team conducted group interviews with parents and SMCs and key informant interviews with head teachers from selected CMP and non-CMP schools. Where appropriate, the team conducted key informant interviews with EMO school managers. School-level data informed Questions 1, 2, and 3. At the district and provincial levels, the team conducted key informant interviews with select stakeholders and decision-makers. Stakeholders identified by the team included relevant provincial-level staff within the SELD, including the offices of the RSU, Project Management and Implementation Unit, and PPP Node; DEOs and taluka education officers (TEOs); and district-level EMO staff. District- and provincial-level data informed Question 3. Key informant interviews with CMP implementing partners informed the assessment of CMP support for policy reforms under Question 3 and the assessment of indicators and targets under Question 4. Other Secondary Data To establish CMP’s operating context, scope of CMP activities, and reported progress toward achieving program outcomes, the evaluation team conducted a structured review of program documents and other relevant secondary data, including quarterly and annual reports; meeting minutes of the CMP advisory group, program steering committee, DSTs, SMCs, and other committees; proceedings of social mobilization sessions; DST terms of reference; EMO concession documents; education sector policy notifications; and other baseline studies, assessments, reports, and databases (Annex 6). Sampling for School-Level Primary Data Collection CMP community engagement interventions occurred primarily at the school level. Thus schools (or school catchment areas) constituted the evaluation’s primary unit of analysis. For school-level primary data collection, the evaluation team sampled 21 CMP schools and 3 non-CMP schools for comparison.8 The 24 total schools in the sample included 22 schools across 6 districts in northern Sindh and 2 schools from among the 5 towns of Karachi where CMP activities were implemented. 9 The 21 CMP schools in the sample were selected according to criteria designed to capture the full range of CMP interventions. The sample included 5 schools awarded EMO contracts and 16 randomly selected construction and neighboring schools. The sample ensured representation of the following: • Boys’, girls’, and mixed schools; • Primary, middle, and secondary/high-secondary schools; • Neighboring and campus (construction) schools; and 8 Comparison schools were randomly selected from a sample of non-CMP schools from CMP districts and with characteristics similar to CMP schools, matched using 2012–2013 SEMIS data. Matching criteria included variables related to infrastructure, school facilities, student/teacher ratios, and location. However, the evaluation was not able to control for other donor activities in non-CMP schools (see Annex 7). 9 For security reasons, one intervention district (Jacobabad) was excluded from the sample. The sample ensured distribution of schools across each of the other six districts and Karachi. Sindh Community Mobilization Program – Midterm Performance Evaluation 8 • High- and low-performing schools with respect to enrollment growth. 10 At each school, the evaluation team interviewed the school head teacher, a group of parents randomly selected based on the enrollment status of their children, and members of the school’s SMC.11 Annex 7 summarizes the distribution of sampled schools by district and intervention type and provides a more detailed sample distribution of schools by sex, grade level, and enrollment growth trends. Data Analysis Methods The evaluation team employed quantitative and qualitative analysis methods to triangulate the data sources described above. These methods included the following: Quantitative Analysis The quantitative component of the evaluation’s analysis estimates CMP’s effect on regressed enrollment trends for boys and girls in CMP districts. To generate these estimates, the evaluation team used SEMIS data from 2012–2013 to generate a matching sample of non-CMP schools located in CMP districts and with characteristics similar to CMP schools. The matched sample allowed the evaluation team to control for other variables that may affect enrollment, such as school infrastructure and student/teacher ratios. A comparison of changes in enrollment in CMP schools and their matching schools produced estimates of change in enrollment rates attributable to CMP (Annex 9). Qualitative Analysis Beyond quantitative estimates of CMP’s impact on changes in enrollment, the analysis predominantly consists of content analysis of qualitative data. Qualitative data from group and key informant interviews helps to explain quantitative findings in Question 1 and is the primary data source informing Questions 2 and 3. The qualitative analysis distinguishes school- and district-level findings from provincial-level findings. At the school level, the evaluation team identified thematic patterns across 24 sets of school￾level group and key informant interviews to explain how and why factors contributed to outcomes of interest. At the provincial level, the evaluation relies on analysis of informed insights from key informants at relevant institutions. Generally, where there is homogeneity or commonality across respondents (for example, in group interviews with parents or SMCs or in key informant interviews with DEOs), the analysis aggregates and reports the frequency of qualitative responses. The analysis triangulates these frequencies with illustrative responses that clearly explain knowledge, attitudes, or behaviors of interest. The evaluation pays particular attention to how gender affects themes in the qualitative data. Data Limitations Key limitations of the evaluation approach, and methods used to attenuate the effects of these limitations on data validity, included the following: 10 For purposes of this study, school performance was defined by change in school enrollment between 2008–2009 and 2015–2016. Schools in the 20th percentile in terms of increase in gross enrollment, calculated as a percentage of 2008–2009 enrollment, were classified as “high￾performing.” All other schools were classified as “low-performing.” 11 For group interviews with parents, the evaluation team sought to include parents of newly enrolled children, parents of children with sustained enrollment over time, parents of children who had dropped out of school, and parents of children never enrolled. The team relied on school general registers to select parents of enrolled students and dropouts and used convenience sampling to recruit parents of children who had never enrolled in school. Sindh Community Mobilization Program – Midterm Performance Evaluation 9 • Availability, quality, and consistency of SEMIS data: The evaluation’s quantitative analysis was constrained by the availability and reliability of SEMIS data. Without class-level data, for example, the evaluation team was unable to measure changes in enrollment by age group. Changes in reporting procedures and force majeure events (i.e., 2010 floods) further compromised the reliability of SEMIS data over time. To mitigate this limitation, the evaluation team conducted thorough assessments of data quality and incorporated quality deficiencies into its interpretation of results. The evaluation team further supplemented and triangulated estimates of gross enrollment with perception data from qualitative interviews. • Availability of data for other donor activities: The evaluation team did not have access to data for other donor activities that may have affected study outcomes in CMP or non-CMP schools. The team therefore could not control for non-CMP activities to isolate the effect of CMP on those outcomes. Where possible, the evaluation team mitigated this limitation in its primary data collection by identifying such activities in sampled schools. This limitation could not be mitigated in the quantitative analysis of SEMIS data. • Sample size: Given time and budget constraints for primary data collection, the evaluation team selected a number of schools in the evaluation sample (24) to balance breadth (i.e., larger sample, less detailed instrument) with depth (i.e., smaller sample, more detailed instrument). The sample size necessarily imposed limits on the evaluation team’s confidence in the generalizability of the evaluation findings. To address this limitation, the evaluation team employed a largely qualitative approach to answering “how” and “why” questions to ensure depth of response, while strategically sampling schools to capture variation in type of intervention and type of school. The evaluation team also relied on multiple data sources to triangulate and verify findings. • SBEP integration: CMP constitutes one of six components of SBEP, each of which may have had a significant effect on CMP outcomes, such as enrollment. The broad scope and degree of integration of SBEP components limited the extent to which the evaluation team could control for the effect of each component, and thus the extent to which the team could attribute outcomes specifically to community mobilization and engagement activities. To mitigate this limitation, the evaluation team’s sampling strategy purposively selected newly constructed schools and employed a qualitative approach to answer “how” and “why” CMP contributed to outcomes of interest. • Response bias: Key informants and group interview participants were subject to response biases that could influence findings and lead to under- or over-estimations of program impact. Specifically, parents with time and willingness to participate in group interviews may have been systematically more likely to routinely engage with schools and/or CMP and thus may overstate the value of CMP community engagement activities. Conversely, people whose children were not enrolled in schools may have chosen to disengage from the education setting entirely and may either lack familiarity with or understate the value of community engagement activities. To mitigate this bias, the evaluation team relied on multiple data sources and probed suspected respondent biases. • Respondent recall: Key informant interviews relied on informants with credible knowledge of a program’s impact on evaluation outcomes. Organizational turnover or other factors impeding respondent recall may have affected the reliability of key informant data. The evaluation team closely consulted with USAID and CMP’s implementing partners to identify appropriate institutions and individuals with relevant knowledge of CMP. Sindh Community Mobilization Program – Midterm Performance Evaluation 10 FINDINGS Question 1: CMP’s Contribution to Access to Schooling How and to what extent has CMP contributed to access to schooling for boys and girls in target communities? To determine how CMP has contributed to access to schooling for boys and girls in target communities, the evaluation considered the context in which parents make decisions about their children’s schooling. The effectiveness of CMP activities for increasing enrollment, retention, and attendance hinges on the interplay of community engagement with a multitude of household and school-level factors, including opportunity costs, availability of schooling options close to the home, and supply-side factors such as quality of instruction and school facilities. A consistent theme that emerges here and in the context of all four evaluation questions is the importance of supply-side factors in determining the effectiveness of CMP strategies for achieving program objectives. Few parents reported community engagement activities as the primary factors motivating their enrollment decisions. Nevertheless, parents frequently referenced determining factors directly affected by CMP activities, namely their own awareness of the value of education, their motivation to educate their children, and their children’s motivation and interest in attending school. Parents and head teachers alike reported that CMP positively influenced enrollment decisions and that outreach activities played a significant role in motivating parents and students to engage with the school community. Enrollment trends in the SEMIS data reinforce this finding, showing a greater increase in enrollment in CMP schools than in comparable non-CMP schools. Factors Affecting Enrollment and Retention The decision to enroll and keep one’s child in school was an active one. Parents weighed many factors, both internal to the school and internal to the household. How these factors, particularly those that are economic or that derive from gender norms, influenced decision-making changed as children grew older. Key household factors affecting parents’ enrollment decisions included awareness of the importance of education and motivation to educate one’s child, perceived economic costs of education, and proximity of available schooling options to the home. Parents’ awareness of the value of education and their motivation to educate their children emerged as the most commonly reported factors driving enrollment decisions in sampled communities (see Figure 2). Parents’ desire for their children to be educated derived from desires for their children to be good citizens; good mothers, wives, or members of the household in the case of girls; or (as discussed below) good economic earners in the case of boys. In at least three cases, this desire derived from recognition of education as a right or religious obligation. Other frequently cited household factors included the perceived economic costs or benefits of education and physical proximity between home and school. These factors were often closely interlinked. Parents commonly cited transportation costs, for example, which increased with distance between home and school. In addition to short-term economic costs, parents considered the opportunity costs of not sending their children to work and the potential longer-term economic benefits of education. Parents in 18 of the 24 communities felt that education would create future economic opportunities for their children. Conversely, perceived lack of employment opportunities for educated youth served as a deterrent to schooling—in Qambar Shahdadkot, for example, a parent noted that even “the educated people also become laborers.” Beyond cost, proximity between home and school presented a complex set of other exigencies. In addition to transport costs, parents considered the Sindh Community Mobilization Program – Midterm Performance Evaluation 11 availability of transport, the safety of children who would need to travel long distances, and, for girls, the availability of male accompaniment. For boys and girls at an early age, factors affecting enrollment generally did not vary by gender, with one important exception. Cultural norms and attitudes about girls’ education posed an additional impediment to girls’ enrollment (see Figure 2). In 16 of the 24 sampled communities, parents cited social barriers to enrolling their daughters in school, including a need for girls to assist with household chores, fears of permitting greater mobility for girls, general societal opinions of the value of girls’ education, and, according to head teachers from at least 3 communities, parents’ fears of girls intermingling with boys of different castes. Where parents had to choose between a boy’s education and a girl’s, many chose that of the boy. A parent in Sukkur noted, “If I have to choose between educating my daughter or son, then I will choose my son. The reason is that girls’ education is not appreciated in our society.” Such barriers notwithstanding, parents or head teachers in nine communities noted positive cultural changes encouraging parents to enroll their daughters in school. Key supply-side factors internal to the school—including quality of instruction, infrastructure, and school facilities—also affected parents’ enrollment decisions. When deciding whether to enroll their children, parents placed great value on the internal conditions of a school, including both the quality of instruction and the quality of school facilities. Parents from 20 of the 24 schools reported that considerations of teaching quality affected their enrollment decision, including the quality of instruction, number of teachers, teachers’ attendance, and familiarity with teachers as members of their community. FIGURE 2: TOP FACTORS AFFECTING PARENTS’ ENROLLMENT DECISIONS Source: Analysis of MSI interview data. The availability and condition of school facilities were also of primary importance to parents’ enrollment decisions. Parents expressed extreme reticence to send their children to schools that did not have basic amenities such as electricity, running water, furniture, functioning washrooms, or good sanitary Sindh Community Mobilization Program – Midterm Performance Evaluation 12 conditions. Parents in at least seven schools noted lack of space for their children in the classroom.12 It was often the case, particularly for girls, that a school in poor condition was the only schooling option in a community. If necessary, girls used the washrooms of nearby homes, but for cultural and safety reasons this was generally considered inappropriate after fifth grade. The need for adequate school facilities was thus particularly acute for girls. A parent in Karachi described the challenges that girls faced when enrolled in schools of poor condition: “Still there are a number of children in the village who are not enrolled in school. Still parents want to send their children to school, and children also want to come to school, but schools don’t have space. There is no washroom facility in the school … Boys go to mosque washrooms, but girls cannot use the mosque washroom or go to nearby houses. Girl students cannot leave school and return home just to use the washroom. This is a problem for girl students.” Conversely, parents in 11 communities reported that a new school building, or the promise of a new building, had affected or would affect their enrollment decision. In Khairpur, a parent directly linked the availability of proper facilities with girls’ enrollment, noting, “After the construction of the new school building, the enrollment of female students has increased. Now girls come in the evening shift too, now they know the importance of education.” As children grew older, the perceived costs and benefits of education changed. Factors affecting girls’ and boys’ retention diverged. As children grew older, the perceived economic costs of and social barriers to education increased. Parents’ awareness and motivation to educate their children was less important to their decision to keep their children enrolled in school than to their initial enrollment decision (Figure 3). The reasons differed for boys and girls. As boys grew older, the expectation that they should work or contribute financially to the household began to compete with and often outweighed the perceived need for continued education. As girls grew older, cultural barriers to continuing schooling intensified. For both boys and girls, supply-side factors—fewer middle and secondary schools than primary schools and higher transport costs associated with reaching more distant middle and secondary schools—were key considerations underlying retention decisions after primary school. For boys, the expectation to work or contribute financially to the household was a key factor affecting retention. Group interview responses indicated that opportunity costs were likely to affect enrollment decisions for boys once the boys were old enough to earn an income (Figure 3). Parents in 13 communities indicated that the need for children to work affected enrollment decisions. In 11 of those communities, parents noted that this calculus changed once boys reached a certain age, at which point, as a parent in a community in Qambar Shahdadkot described, “Parents prefer that their boys earn 200–300 [rupees] daily instead of going to school.” A parent in a second community in Qambar Shahdadkot illustrated the process by which parents weigh the immediate costs of education with expected future benefits, noting, “I asked my children to help me in grazing the animals. I have seen many educated persons who were unable to get jobs. It is better that children should start working at an early age.” Cultural norms around the acceptability of leaving school to work contributed to this calculus. A parent in a third community in Qambar Shahdadkot painted the following picture regarding his son’s response to corporal punishment: “Once his teacher punished him and he felt insulted. I asked my son to go 12 The issue of space to accommodate new enrollment is discussed further in Question 4, where the appropriateness of the enrollment indicator is considered in the context of overenrolled schools. Sindh Community Mobilization Program – Midterm Performance Evaluation 13 school, but he lost his interest and is now working in the field. We are farmers and have animals … He supports me.” Compared with boys, girls were more likely to be kept out of school to assist with household chores. While the expectation to generate income was lower for girls, it was not altogether absent. Parents in three communities described the expectation to work as a factor affecting girls’ enrollment, retention, or attendance. This expectation, and its effect on girls’ schooling, may be seasonal. For example, a parent in a community in Sukkur noted, “My daughter remains absent because she is working in the cotton fields. It is her grandfather’s responsibility to take her to the school, and he will continue to do so when the cotton season ends.” In other cases, such as in a community in Dadu, girls engaged in stitching, handicrafts, or embroidery in the home to contribute to household income. For girls, socio-cultural barriers were key factors affecting retention. Socio-cultural barriers, including community attitudes about girls’ education, expectations for girls to marry, and concerns about girls’ mobility, weighed more heavily on retention decisions as girls grew older (Figure 3). Parents in 16 communities cited such barriers to girls’ continued education. In most cases (14 of the 16 communities), these barriers stemmed from traditional thinking about the age at which girls were considered grown adults, the appropriate age for marriage, the value of girls’ education, and what constituted the appropriate environment for girls. In other cases, these barriers were rooted in broader community dynamics and included parents’ perceived risks to the safety and security of their daughters. In a community in Qambar Shahdadkot, for example, a parent noted: “The social setting of this village does not encourage females’ education … In our village different castes are living together … We are afraid that [boys from other castes] will harass our girls. If that happens, there will be conflicts among [the] elders.” The effect of socio-cultural barriers on girls’ retention cannot be isolated from other factors. Community attitudes about girls’ education were particularly compounded by the lack of easy access to middle and secondary schools—in six communities, parents reported there were no schools for their daughters near their homes. A parent’s objections to mixed-sex education may further restrict schooling options. One parent in Karachi explained, “When [girls] reach fifth grade, we as parents want our girls to be taught separately from boys by female teachers. Most of the parents stop sending their girls to school because the girls have grown up. Due to social norms, it is necessary that those girls should be separated from boys.” Availability of secondary schooling options was severely limited in target communities. Proximity of home to secondary schools thus became a driving factor for retention of both boys and girls after primary school. The immediate accessibility to schooling diminished significantly after primary school. According to SEMIS census data, 73 percent of enrolled students in Sindh were of primary school age. At the same time, of the 45,447 government schools in Sindh, 41,131 (almost 91 percent) were primary schools.13 This meant that even without accounting for the drop in enrollment after primary school, the 27 percent of enrolled students that were of post-primary school age were accessing just under 10 percent 13 Secondary schools were typically larger than primary schools (according to SEMIS 2015–2016 data, primary schools had on average two classrooms per school, compared to nine classrooms per school in non-primary schools). However, on average, there was little variation in students per classroom between primary and non-primary schools (35 students per classroom in primary schools, 38 students per classroom in non-primary schools). Sindh Community Mobilization Program – Midterm Performance Evaluation 14 of the schools in Sindh.14 In the seven districts and five towns of Karachi where CMP was active, SEMIS data indicated a total of 12,552 primary schools, compared with 1,379 higher-level schools. As a result, children generally needed to travel further to reach secondary schools than primary schools. Parents in 16 of the 24 sampled communities cited distance between home and secondary schools as general impediments to keeping children in school. In 10 of those communities, parents reported that transportation was not available, while parents in 14 communities cited transport costs as an impediment to continued education. For reasons discussed above, proximity between home and school was a particularly decisive factor for girls after primary school. A group interview participant in Ghazi Markhan explained the predicament of sending her daughter to school: “Girls’ middle school is not functional in the village. Only a few girls study in the boys’ middle school. The majority of girls who pass primary do not get admission to middle school because of the mobility issue. The female students must either go with their father or the neighbor’s daughter who is also enrolled in the same school. Parents do not have time to provide pick-up and drop-off to their daughters.” Due to these impediments, parents commonly reported that their children would stay in school if the local primary school were upgraded to a secondary school. Parents in four communities were explicit that upgradation had affected or would affect their enrollment decisions, and parents in five communities felt that the construction of a new school had affected or would affect their enrollment decisions. FIGURE 3: COMPARISON OF FACTORS AFFECTING PARENTS’ ENROLLMENT AND RETENTION DECISIONS Source: Analysis of MSI interview data. CMP’s Contribution to Enrollment and Retention Decisions Against this backdrop of socio-economic and cultural factors, the effect of CMP’s community mobilization and engagement efforts on enrollment in CMP communities was well-recognized by parents, head teachers, and SMCs. Parents were most likely to recognize CMP’s role in raising 14 The Pakistan Social and Living Standards Measurement Survey (PSLM) conducted by the Pakistan Bureau of Statistics estimated that the gross enrollment rate for boys and girls at the primary level was 61 percent in 2014–2015. At the middle school level, it fell to 34 percent, and at the matric level, it fell again to 25 percent. This reinforces a supply-side constraint to retention—higher secondary enrollment rates would place further strain on available secondary schools. See PSLM 2014–2015. Sindh Community Mobilization Program – Midterm Performance Evaluation 15 awareness of education processes and the value of education, and in disseminating information about school affairs. Head teachers and SMCs were most likely to recognize CMP’s role in coordinating community engagement and mobilization activities. Head teachers also called attention to CMP’s capacity-building interventions with SMCs and SMC subcommittees. Generally, CMP’s efforts coalesced to make a strong contribution to the socialization of education, including for girls, in target communities. CMP increased parents’ awareness of the value of education and broadened engagement efforts to include a focus on girls’ education. Parents most commonly felt that CMP’s contribution to enrollment resided in building awareness about the education process. Parents in eight communities cited CMP’s contribution to their knowledge of the education process through community outreach and the dissemination of information about enrollment and (in at least three cases) health and hygiene. SMC members, who constituted CMP’s primary partners for community engagement and mobilization, were likewise most likely to highlight the value of community engagement events and their effectiveness in raising awareness about enrollment. SMCs in nine communities cited CMP’s contribution to enrollment through coordination of outreach activities and increased community awareness. An SMC member in Kashmore drew a direct link from community outreach to awareness and then enrollment: “We meet with the parents who attend the functions and we motivate them to put their child in our school. Through these walks and rallies, parents gain awareness about education. Parents see that the programs are focused on education. When they see teachers and [the deputy commissioner] all joined together to [participate in a] walk, they realize there is importance in education.” At the same time, the evidence shows that CMP broadened engagement efforts to include a greater focus on girls’ education. In four communities, parents recognized an attention shift to girls’ enrollment. In three of those four communities, parents pointed to female mobilizers who engaged specifically with women in the community. A parent in Kashmore explained: “They push girls’ education. Our house is near to the school, so they call us for meetings regarding educating the children. The Wadera asks us to get our girls enrolled. Females visit our houses; the peon brings them to our homes to talk about our girls’ education.” According to an SMC member in Khairpur, direct engagement of women in the community increased CMP’s effect on enrollment. He noted, “Parents started sending their children. As they communicated with the families, especially females, enrollment improved by about 25 percent.” CMP community engagement and mobilization activities in target communities helped to socialize the notion that education was accessible and valuable. A driving force behind parents’ decisions about their children’s education in CMP communities was trust—trust in head teachers, teachers, and other students and their families. By engaging parents in the education process, CMP helped to socialize the value of education in target communities. Socialization occurred in three ways. Most prominently, enrollment campaigns and other community events increased parents’ and students’ desire to participate in school activities. Parents, head teachers, or SMC members in five communities reported that community events sparked interest in schooling among parents or children inspired by a desire to share in the experiences of the larger community. An SMC member in Larkana, for example, explained that the distribution of prizes at community events increased children’s interest and “has impact on those students who are not coming to school.” Likewise, a head teacher in Khairpur noted that “CMP helped us to regularize the students by distributing prizes to the children. This motivated and created interest from other children to be regular in class.” Sindh Community Mobilization Program – Midterm Performance Evaluation 16 Second, the improvement of school facilities through small grants and engagement of taluka- and district￾level officials lent importance to the school as a community institution. In Karachi, for example, an SMC member illustrated how school improvements incentivized parents to pressure both the school and district officials to upgrade the school: “Due to the small grant scheme, the condition of the school improved. The parents who were reluctant to send their children regularly due to the unavailability of electrical fans and water were satisfied with school conditions. Now the community is pressuring the head teacher and SMC members to upgrade the school to secondary level. The District Education Office has also realized the need for upgradation and asked the head teacher informally to start grade six. Currently 36 children are studying in sixth grade without notification of upgrading.” Finally, SMC subcommittees, established with CMP support, broadened the scope of school interaction with parents, linking enrollment and education with other basic household needs, such as health/hygiene and disaster response. CMP small grants reduced the likelihood that poor school conditions would deter parents from enrolling their children. As discussed, inadequate school facilities and lack of proper amenities affected enrollment. The evaluation found qualitative evidence that CMP school improvement interventions—including small grants and technical assistance and capacity building for school financial management—positively affected enrollment. Parents in nine communities reported that improvements to school conditions affected their enrollment decisions, which in some cases meant transferring children from private to government schools. A parent in Karachi explained, “The SMC and head teacher listened to our problems and they have resolved the issues of drinking water, electricity, and furniture. With the improvement of school facilities, parents have transferred their children from private schools.” Likewise, head teachers and SMC members in 10 communities attributed increases in school enrollment at least in part to CMP’s efforts to improve facilities, support school maintenance, or strengthen SMCs. CMP’s Estimated Effect on Enrollment Using SEMIS data, the evaluation team employed a quasi-experimental interrupted time series approach to estimate the impact of CMP on growth in overall enrollment. To directly attribute enrollment growth to CMP, the evaluation team used propensity score matching to construct a group of comparable non￾CMP schools (i.e., a theoretical counterfactual) for comparison. The team then measured differences in enrollment trends between CMP and non-CMP schools to isolate changes in enrollment attributable to CMP. For details of the matching approach and interrupted time series analysis, see Annex 9. CMP schools increased overall enrollment at a significantly higher rate than non-CMP schools. Across all schools, the evaluation estimated the total increase in enrollment attributable to CMP to be 7,882 students. Prior to CMP’s intervention, the evaluation team found an upward growth trend in enrollment for both CMP and non-CMP schools. The results show that total enrollment in CMP schools was statistically higher than in non-CMP schools at the beginning of the time series, i.e., 2008–2009, but that CMP and non-CMP schools grew at comparable rates during the pre-CMP period. The data indicates that CMP schools had a significantly higher growth trajectory than non-CMP schools following the intervention. Figure 4 summarizes the estimated contribution of CMP to enrollment for boys and girls in various types of schools. The data show no significant difference in enrollment between CMP and non-CMP primary schools for boys or girls, but show a significant increase in enrollment rates for both boys and girls in Sindh Community Mobilization Program – Midterm Performance Evaluation 17 CMP middle and secondary schools (which in many cases also include primary grades) relative to non￾CMP schools. Overall, the evaluation estimates a total increase in enrollment of 7,882 students attributable to CMP (Figure 4).15 FIGURE 4: SUMMARY OF CMP’S CONTRIBUTION TO ENROLLMENT Source: Analysis of SEMIS data. Note: Red represents statistically significant changes in enrollment. Blue is not statistically significant. New construction and overenrollment mitigated CMP’s contribution to increased access to schooling for boys and girls. Group interview data validate the SEMIS findings regarding CMP’s contribution to enrollment. Head teachers in 15 of the 21 CMP schools and 2 of the 3 control schools reported that enrollment had increased in their schools in the last two years. Group interview data also shed light, however, on factors mitigating CMP’s contribution to enrollment. 16 The chief factor mitigating CMP’s effectiveness in increasing enrollment is overenrollment, i.e., the lack of physical capacity of CMP schools to accommodate new enrollment. All 15 schools that reported increases in enrollment also reported associated management challenges. Thirteen head teachers reported insufficient space or inadequate facilities to accommodate additional enrollment, and nine head teachers reported too few teachers. 15 In October 2016, SBEP commissioned a study to compare enrollment trends in CMP and non-CMP schools, titled “Data Portrayal of Increase/Decrease in Enrolment, particularly of Girls’ [sic] Students in CMP Schools – Trend Analysis.” This evaluation provides an alternative approach to estimating CMP’s contribution to enrollment. This approach deviates from that of the 2016 trend analysis in two ways: (1) For purposes of attribution, the evaluation identified a comparison group of schools using propensity score matching (rather than a more limited set of control criteria), thus controlling for additional confounding factors likely to influence enrollment, including school facilities, and minimizing differences between CMP and matching schools; and (2) the evaluation used regression analysis (rather than a before-and-after comparison) to estimate enrollment trends of CMP and non-CMP schools before and after CMP, thus accounting for the variability in the enrollment data. Whereas the 2016 trend analysis concluded that CMP increased enrollment by 9,521 students, the evaluation’s revised approach yielded a significantly lower estimate of CMP’s impact on enrollment. 16 These mitigating factors have implications for the appropriateness of enrollment as an indicator and relevance of current CMP enrollment targets. The findings for Question 4 discuss these implications in greater detail. Sindh Community Mobilization Program – Midterm Performance Evaluation 18 Some schools have addressed space challenges by splitting the school day into two shifts, creating challenges for student attendance. A head teacher in Qambar Shahdadkot explained: “This school is running in the evening shift and it is difficult for students to attend… It is particularly difficult for girls because they must do household chores in the evening. Similarly, girls must come along with their fathers or brothers, and most of them are not available in the afternoon. They must go for work in the morning.” A second mitigating factor, at least in the immediate term, is new construction to accommodate more students. During the construction phase, parents may need to shift their children to temporary schooling arrangements with inadequate space or amenities. Commonly, students attend alternative schools that require longer travel to unfamiliar environments or that can only provide schooling in the evening shift. In at least two cases, one in Larkana and one in Sukkur, host schools accommodated female students from construction schools only during the evening shift, adversely affecting girls’ enrollment and attendance. CMP community engagement activities in temporary schooling arrangements were also limited. Conclusion Powerful factors—both supply-side factors internal to the school and demand-side factors internal to the household—had significant bearing on parents’ decision-making about their children’s enrollment and retention. These factors, and the concomitant perceived costs of education, intensified as children grew older. Nonetheless, CMP-sponsored activities increased awareness and motivated parents to overcome these factors. More broadly, CMP engagement activities contributed to community conceptions of the social importance and value of education. While supply-side challenges remained a strong hindrance to enrollment and even more so to retention, CMP small grant interventions and contributions to schools’ financial management made some progress in removing that deterrent. Quantitative data and respondent feedback from parents, head teachers, and SMCs told a positive story about CMP’s contribution to access to schooling, including CMP’s specific contribution to girls’ access. Though it is difficult to isolate the impact of engagement and mobilization activities in the context of other economic and socio-cultural factors, as well as other SBEP interventions, the SEMIS data make a strong case that CMP schools experienced greater enrollment growth as a result of CMP interventions. Question 2: Effectiveness of CMP Community Engagement Activities To what extent have CMP processes been effective for engaging target communities? CMP engagement and supply-side interventions were implemented in close partnership with SMCs, suggesting a positive correlation between a strong SMC partner and positive enrollment outcomes. The evaluation found SMCs to be effective vehicles for community engagement. CMP demonstrably strengthened SMCs in target communities. Despite persistent challenges to SMC engagement of communities, CMP-sponsored community activities increased SMC and school interaction with parents, catalyzed community and school interaction with DEOs and TEOs, and, in some cases, mobilized public sector resources for school improvement. Role of SMCs in Community Engagement SMCs, as government notified entities formed in every school “for better management of the institution through community participation and shared accountability,”17 represented a unique and vital 17 See RSU, http://www.rsu-sindh.gov.pk/units/SMC.php. Sindh Community Mobilization Program – Midterm Performance Evaluation 19 institutional link between parents and schools and a natural institutional partner for CMP’s community engagement and mobilization activities. Three factors buttressed SMCs’ value as CMP partners: (1) SMCs’ representative membership structure, (2) SMCs’ formal mechanisms for school-level budgetary spending, and (3) the considerable degree of government ownership of SMCs. Challenges for SMCs as primary and sustainable mobilization agents included parents’ willingness (or lack thereof) to engage, community distrust of SMCs, and lack of female representation on SMC executive bodies. SMCs’ representative membership structure provided an official forum for communication with parents and for parents’ participation in school affairs. By law, the SMC executive body is comprised of a secretary (school head teacher), elected chairperson (typically a parent or influential member of the community), and three elected members (also typically parents).18 The broader SMC general body includes all parents or guardians of enrolled students. Evidence shows that where SMCs were active, parents actively participated in SMC activities. Parents in at least 10 of the 21 group interviews reported having participated in SMC executive or general body meetings, SMC community events, or SMC subcommittee activities. In 10 communities, parents reported informal interaction concerning school activities with head teachers or other SMC members outside general body meetings. A parent in Khairpur described his experience with an active SMC: “The SMC members brief us about the school matters and then we discuss it with the head teacher … We have also been consulted for the new building and discussed new school building renovations and construction of washrooms in the new building … We have discussed with the head teacher the need for washrooms for younger girls … SMC members taught us how to work together as a team.” SMCs provide formal mechanisms for budgetary spending. SMCs were designated recipients of government funds for school maintenance and repair, and thus represented the core school-level mechanism for supply-side activities. As such, SMCs were a natural conduit for community participation in school improvement processes. According to IRD, prior to CMP’s launch, 20 of the 21 intervention schools had official SMC bank accounts to receive government funds for school improvements. According to one education official in Larkana, the competent and transparent use of SMC funds could inspire community confidence and trust in school management. SMCs enjoyed a strong degree of ownership from GoS institutions. CMP anchored its capacity-building and technical assistance interventions within existing SMC structures mandated and supported by the GoS. Evidence suggests that SMCs garnered a strong degree of ownership from GoS institutions. Each of the three DEOs interviewed for this evaluation, for example, recognized SMCs as the principal entity responsible for community engagement and mobilization. An official at RSU indicated the same, noting, “In terms of community participation … [school management] committees are working as per defined criteria and they have a vital role with the community.” Having developed the SMC regulatory framework and allocated funding to SMC activities, RSU indicated its commitment to supporting the SMC, arguing that funds had been released and “government has notified the rules and regulations of SMCs, so we want now for [this year’s] election process to commence.” The primary challenge to SMC community engagement and mobilization activities was lack of parental interest in or commitment to school affairs. Key among the challenges to engagement cited by SMCs was a lack of parental motivation to engage. SMC members in seven communities cited difficulties motivating parents or keeping parents motivated 18 By law, SMCs should also include two student representatives. Sindh Community Mobilization Program – Midterm Performance Evaluation 20 over time. SMC members in seven communities cited resource constraints—including financial, time, and transport—that precluded parents from participating or engaging with SMCs. SMC members in three communities called attention to community distrust in SMCs as an institution, partly due to lack of understanding of the SMCs’ responsibilities, but also due to distrust in SMCs’ management of school funds. An SMC member in Qambar Shahdadkot described his predicament, noting, “Earlier, people were not clear about the role of the SMC and they had misconceptions that the head teacher and SMC chairperson were taking SMC funds for their own benefits. CMP clarified the role of the SMC …” This source of distrust had important implications for CMP’s engagement strategies, insofar as CMP worked to build transparency into SMC processes and clarify roles and responsibilities for both SMC executive bodies and SMC general bodies. Pre-existing SMC structures were predominantly male in composition. CMP-led creation and growth of subcommittees had moderate success in increasing female participation in SMC activities. Except in cases where CMP helped constitute new SMCs, the program worked through pre-existing SMC structures and did not work to change SMC membership. SMC members in two communities with exclusively male SMCs expressed a desire to incorporate more women into executive bodies or subcommittees, but in at least four CMP communities, attitudes about women’s involvement precluded their participation. According to an SMC member in a community in Dadu, for example, it was not possible to involve women in SMC activities: “The CMP team cannot meet with women because of the village environment. Also, women cannot be members here. We had a training in Karachi where I met a member of EDLINKS.19 She wanted to visit the school. While I was showing her the school, the villagers started to ask me who this woman was and why was she not wearing an abaya. In rural areas, it is a big problem. They do not accept this.” The typically male composition of SMCs in most CMP intervention schools at times presented challenges to engaging with mothers, and likely limited outreach related to girls’ schooling. In Karachi, for example, parents lamented the lack of women’s participation in SMC activities, noting, “If a female teacher comes, then the community women may be inspired and more likely convinced to join the SMC as a member, which will facilitate the mobilization process.” Conversely, SMCs comprised exclusively of female members (as in the case of a girls’ school in Larkana) faced barriers to engaging men in the community. CMP had moderate success involving more women in SMC subcommittees—in 2016, women made up 25 percent of the membership of 282 subcommittees (Annex 10).20 A community in Dadu, for example, formed separate health subcommittees for men and women. Effect of CMP on SMC Capacity CMP capacity-building and technical assistance efforts had a demonstrable effect on the activeness and strength of SMCs. Although 20 of 21 SMCs in CMP intervention communities had active bank accounts prior to CMP’s intervention, these SMCs were not necessarily active. For example, according to IRD, prior to the launch of CMP, SMC chairpersons and/or general secretaries (i.e., head teachers) actively participated in SMC activities in only 11 of the 21 communities. CMP contributions to SMC capacity recognized by respondents included strengthening of internal processes through clarification of roles and responsibilities; introducing processes for financial management, record keeping, and school improvement planning; forming and strengthening SMC subcommittees; and general capacity building and technical assistance for community outreach, engagement, and mobilization. 19 USAID funded the Links to Learning Program (EDLINKS) in Pakistan from 2007 to 2012. 20 CMP Annual Report, 2016. Sindh Community Mobilization Program – Midterm Performance Evaluation 21 CMP technical assistance clarifying SMC roles and responsibilities extended to both executive and general bodies of target SMCs. Such technical assistance increased parents’ engagement with SMCs. In six school communities, SMCs reported that CMP clarified the SMCs’ roles and responsibilities, including the SMCs’ appropriate role in community engagement and mobilization. An SMC executive body member in Karachi noted that prior to CMP, “SMC members were unclear about their roles and responsibilities. CMP organized regular meetings with [members] and provided short training sessions on the functions, roles, and responsibilities of the SMC.” Notably, technical assistance in this regard extended to the general body. As one member of the same SMC in Karachi explained, the purpose of CMP meetings with parents was in fact “to inform communities about the role of SMCs.” Clarification of roles and responsibilities was particularly useful for new SMC participants. One member elected to the SMC executive body of a school in Dadu noted: “I and other SMC members joined the school after CMP. We were elected … through a general body meeting … Earlier it was not like this. Instead the SMC had only a landlord and a chairperson … Now they have [informed us] about the utilization of funds and everything related to the SMC.” CMP increased SMC capacity to effectively utilize resources by strengthening processes for financial management, record keeping, budgetary planning, and utilization of SMC funds. Perhaps CMP’s most consequential contribution to SMCs’ capacity was the strengthening of SMC processes for financial management, record keeping, and allocation and implementation of SMC funds. Fifteen SMCs reported that CMP helped them to improve their processes for accessing and utilizing SMC funds. CMP technical assistance facilitated SMC compliance with district budgetary requirements and introduced processes for needs identification, community participation in spending decisions, and school improvement planning. Eleven SMCs recognized CMP’s assistance in developing school improvement plans. Activities in support of school improvement processes were particularly impactful for “neighboring” schools, which did not otherwise benefit from construction of new school facilities. CMP technical assistance for financial management broadened participation in SMC spending decisions by helping parents elected to the SMC executive body to participate in SMC decision-making processes, which were previously managed by head teachers. According to an SMC member from a community in Karachi, “The head teacher was active and knew about SMC funds, but the chairman and other community members were initially reluctant to utilize SMC funds. CMP conducted orientation sessions of SMC members to increase their understanding and provided financial management training.” SMC subcommittees broadened community participation in SMC and school affairs. SMC subcommittees tasked with outreach functions specific to social mobilization and school improvement planning, health and nutrition, and local resources and grants enhanced the extent and scope of SMC engagement with parents. Thirteen of 21 SMCs lauded concrete benefits gained from CMP’s efforts to form and strengthen SMC subcommittees. Most commonly, SMC members felt that subcommittees had increased interaction between schools or SMCs and parents, whether through training, community participation in subcommittee activities, or household visits. For example, with reference to the dissemination of remedies for diarrhea, one SMC respondent in Karachi described active subcommittee outreach to communicate health messaging to parents. The respondent noted, “This was very informative for us. We called a meeting and taught this method to [members of the] community, especially community women. We also run hand-washing campaigns and tell communities about the benefits of hand-washing.” Sindh Community Mobilization Program – Midterm Performance Evaluation 22 In five communities, SMC members cited increased community awareness, among both parents and children, of health and hygiene risks as a direct result of CMP activities. An SMC member from a school in Qambar Shahdadkot, for example, argued that “[the] committee creates awareness among children about the importance of washing hands three times with soap, brushing teeth, keeping clothes clean, and trimming nails.” However, unlike SMC executive bodies, SMC subcommittees were not government￾notified bodies, a status which potentially compromised their sustainability. CMP technical assistance built SMCs’ general capacity for community mobilization and spread awareness of SMCs’ role as community mobilizers. In five school communities, SMC members reported increased capacity to engage with and mobilize communities as a result of CMP activities. CMP built SMCs’ mobilization capacity in two ways. First, CMP provided guidance on the appropriate role SMCs can play in community engagement and mobilization; and second, CMP introduced tools and best practices. An SMC member in Khairpur, for example, illustrated the combination of instruction and materials offered by CMP, noting, “CMP provided guidance on how to deal with parents and the role of the SMC in helping parents to enroll their children in school. CMP provided information about procedures to motivate parents, enroll children, follow up with parents if a child is not appearing in school, and bring the child back to school.” CMP commonly supported SMCs by exercising its convening authority, directly coordinating enrollment drives, community events, and general body meetings. Eleven SMCs reported receiving coordination support from CMP. This support produced direct benefits. According to at least 7 of those 11 SMCs, enrollment drives and community events increased student interest, parental awareness, and enrollment. CMP’s coordination function entailed a capacity-building component as well, to the extent that CMP organized such events in partnership with SMCs. As an SMC member in Khairpur explained, “CMP cannot work separately because the community will not respond … We must work collectively. CMP members don’t know about the community. SMC members take them to [the] community.” Effect of SMC Capacity Building on Community Interaction Parents, SMCs, and head teachers reflected positively on the effectiveness of CMP strategies for increasing interaction among schools, communities, and education officials. Parent, SMC, and head teacher respondents reported increased parental involvement in SMC activities and community events and increased formal and informal interaction between parents and SMCs and between schools and DEOs and TEOs. CMP strategies to engage SMCs increased interaction between communities and schools. It is difficult to ascertain the level of interaction that parents had with schools and SMCs prior to CMP’s intervention. According to IRD, prior to CMP, the general body had participated in SMC activities in 3 of the 21 sample schools, and SMCs had led one or more community events in 4 of the 21 communities. At the time of the midterm evaluation, the general body was active in SMC activities in all 21 CMP communities, and SMCs had led community events in all 21 communities. Since the launch of CMP, 11 of the 21 communities had conducted resource mobilization efforts to solicit land, money, and other in￾kind contributions from community members. Evidence from parents reinforced the finding that CMP increased the responsiveness and inclusiveness of SMCs in target communities. In eight CMP communities, parents reported increased participation in SMC general body activities, including development of school improvement plans. Parents in five communities reported increased exchange of information with SMCs, increased confidence in SMCs, or increased responsiveness of SMCs since CMP activities began. Likewise, 11 SMCs reported increased interaction with parents, including greater involvement of parents in SMC activities and subcommittees Sindh Community Mobilization Program – Midterm Performance Evaluation 23 and more formal and informal meetings with parents. By contrast, no parents in the three non-CMP comparison communities reported any formal interaction with their school SMCs. At the same time, head teachers, as members of the SMC executive body and often influential figures in the community, remained an important, if not primary, conduit for interaction between parents and SMCs. According to an SMC member in Karachi, by the time of the midterm evaluation, SMC members in his community had regular interactions with parents, head teachers, and CMP, and the interactions had yielded positive results: “SMC members regularly meet with parents. SMC members meet CMP staff at least every month, [at which time] issues of community participation are also discussed. The head teacher and SMC members meet almost every day. Formal SMC meetings also take place after every two months. In addition to formal meetings, SMC members meet informally whenever there is need. For example, the SMC called parents for submission of a small grant application to CMP. In this meeting, the SMC and parents together prioritized lack of drinking water, furniture, and electric fans as major issues of the school.” CMP community engagement strategies increased community interaction with notables and education officials, creating opportunities to mobilize resources for school improvement and upgradation. Finally, the effectiveness of SMCs for community engagement extended beyond the SMC general body. In addition to engaging parents, SMCs engaged other community organizations and influential members of their respective school communities. At a CMP school in Karachi, for example, the SMC coordinated enrollment efforts with local associations representing each of the Bhand Biradri, Khaskheli Biradari, and Kachi communities. The head teacher of a school in Khairpur drew attention to how engaging village notables affected parents’ participation in community events, noting, “We are organizing a seminar next month and the topic will be ‘Education for Everyone.’ We plan to invite village notables, politicians, local community leaders like the nazim, and community members. This has a positive impact on parents.” Activities meant to increase enrollment through school improvement or upgradation required additional support from district and taluka education offices, as these officials represented the first point of entry for channeling government funds for school improvement and securing notification of secondary school status. CMP community events effectively engaged education officials. Five SMCs reported increased visits to their communities from DEOs and TEOs. In three communities, SMC members reported increased DEO or TEO awareness of school affairs, improved information sharing, or increased SMC confidence to engage with government officials. In one community in Karachi, SMC members attributed a DEO’s decision to upgrade the school to this interaction: “Before CMP, only the supervisor visited the school … However, after CMP, high-level officers, including the DEO and provincial-level officers, visited the school and listened to the school’s problems. Officials also participated in the enrollment campaign and walk organized by CMP. As a result, government officials gave their informal approval to start sixth grade … SMC members now communicate with officials with more confidence.” Education officials, for the most part, echoed these perceptions of CMP’s contributions to community engagement and mobilization. Four of five district and taluka officials interviewed reported participating in CMP activities (such as training sessions and community events) or collaborating with CMP; four of five officials reported that CMP had a positive effect on community mobilization; and four of five officials reported that CMP had a positive effect on SMC strengthening. District-level support for CMP was not uniform, however, a finding which will be discussed in greater depth in Question 3. Sindh Community Mobilization Program – Midterm Performance Evaluation 24 Conclusion SMCs have an indispensable role to play in community engagement and mobilization in their capacity as participatory forums for interaction between parents and schools and as entities with assigned budgetary authority and buy-in from government stakeholders. CMP technical assistance and convening authority have strengthened the capacity of SMCs to engage parents, manage school improvement funds, and engage with government education officials in target communities. To some extent, CMP has improved women’s participation in SMC activities through the formation and support of SMC subcommittees. These efforts have resulted in greater parental participation in school affairs and confidence in school management. Still, barriers to SMCs’ fulfillment of their intended roles and responsibilities persist. Evidence suggests a degree of community distrust of SMCs and a reticence of newly elected SMC members to assume unfamiliar tasks. To optimize SMCs’ performance as vehicles for community engagement and sustain CMP’s engagement strategies, SMCs must ensure that members of the general body fully understand SMC roles, responsibilities, and processes, and strategically engage government education officials in SMC activities. Question 3: CMP Support for Education Sector Reforms To what extent have CMP’s strategies in support of GoS education reforms, specifically school consolidation and public-private partnerships, been effective for achieving GoS reform objectives? Question 3 looks at CMP’s support for GoS education sector reforms, including private sector education service delivery under the PPP Act of 2010 and the School Consolidation Policy of 2012. The evaluation team analyzed support for these reforms at two levels: • At the provincial level, CMP provided provincial officials with technical assistance to manage reform processes. Provincial support for the transition to private sector education service provision under the PPP Act centered on support for the GoS PPP Node. CMP support for the School Consolidation Policy, on the other hand, centered on support for the GoS RSU. • At the district and school levels, CMP conducted community engagement activities to sensitize communities to reforms and provided institutional support during reform implementation. According to IRD, CMP’s support at the local level specific to sector reforms centered around relationship building and coordination among reform stakeholders, including EMO representatives, district administration, DEOs, teachers, SMCs, and parents. This section presents findings first at the provincial level, and then at the district and school levels. CMP Support for Provincial Education Sector Reforms At the provincial level, the evaluation found mixed results regarding the extent of CMP support to provincial units managing sector reforms. CMP support was particularly integral to the PPP Node’s performance, whereas support to the RSU for consolidation was need-based and more pronounced at the policy level than at an institutional level. Where administered, CMP support was effective. Sindh Community Mobilization Program – Midterm Performance Evaluation 25 The PPP Node relied heavily on CMP support for resources, technical guidance, and stakeholder engagement, primarily in the pre-transition phase. At the time of the midterm evaluation, the PPP Node was understaffed and under-resourced. Schools were moving quickly to the EMO model, compounding strains on the node’s capacity.21 According to IRD, CMP’s support for the PPP Node consisted primarily of support for the pre-transition phase and EMO contract administration. CMP assisted the PPP Node in securing financial allocations for the reform from the GoS, defining stakeholder roles and responsibilities, facilitating inter- and intra￾departmental coordination, and ensuring compliance with PPP Act provisions. As evidenced by the successful handover of nine schools to EMO management after the first two requests for proposals (RFPs), CMP provided valuable technical guidance to the PPP Node to manage the EMO transition. The PPP Node called attention to CMP assistance in developing procedures for hiring, procurement, and issuing notifications. EMO managers recognized CMP’s convening authority with PPP stakeholders, authority which the PPP Node lacked due to resource constraints. According to one EMO manager: “The government doesn’t have the capacity to run [the PPP process] on its own … New stakeholders are also coming here. Not every partner is willing to [work with] other stakeholders … Not every partner is interested in working together. CMP is the only mediator who is in coordination with all stakeholders, and if that mediator is taken out, it is not going to work for long.” Specifically, CMP convened DSTs in Sukkur and Khairpur that served as effective mechanisms for information exchange and resolving EMO issues at the district level. EMO managers from both the Institute of Business Administration (IBA) and Charter for Compassion (CFC), the two EMOs that received contracts under the first RFP, recognized DSTs as useful forums for information exchange and coordinating with DEOs and other government officials, which was often necessary to fulfill basic school management functions such as ensuring the appointment and oversight of government teachers. Coordination with DEOs and other stakeholders was also necessary to ensure the sustainability of the EMO model. One EMO manager in Sukkur summed up the importance of securing stakeholder ownership of the EMO process: “These head teachers must own the school. We are here for a short time to manage schools … These are government policies but if we don’t implement the policies effectively from the beginning, then the gaps will last until the end. These loopholes need to be addressed … The system cannot depend on only one person. The DEO must take the teachers on board with coordination of this project to realize how effective these activities can be for the education system in the future.” Ultimately, however, institutionalization and sustainability of the EMO model also require greater empowerment of the PPP Node as the core government entity responsible for the model’s implementation. At the time of the midterm evaluation, CMP had positioned itself well to facilitate sharing of knowledge and best practices with the PPP Node. A PPP Node official described his perception of the node’s partnership with CMP: “I never consider CMP to be a different part. Whenever I have any work I visit their office. I consider CMP’s office as my own office. I consider them as my staff members… We are one unit.” 21 The transition to EMO management included 4 schools in the first RFP (management of which were awarded to IBA and CFC), 5 schools in the second, and 14 in the third. At the time of the midterm evaluation, the GoS intended to transition all 106 SBEP construction schools to EMOs, while also introducing a cluster-based model for EMO management. Sindh Community Mobilization Program – Midterm Performance Evaluation 26 In contrast to CMP’s regular and institutionalized support for the PPP Node, CMP support for consolidation at the RSU has been need-based and guided by specific policy needs. The RSU is responsible for policy formation and oversight of the School Consolidation Policy and serves as the provincial government counterpart for CMP’s consolidation support. According to IRD, CMP primarily provided advisory support to RSU, including consultation on the implementation status of consolidation policies, challenges to implementation, and implementation strategies. CMP’s assistance targeted RSU, but also included the GoS Program Management and Implementation Unit and the SBEP Program Steering Committee. Specific achievements highlighted by IRD included consultation in the drafting of a 2015 Addendum to the School Consolidation Policy and the dissemination of a SELD notification delegating administrative and financial powers to campus head teachers and principals. In two districts, Larkana and Qambar Shahdadkot, the Program Management and Implementation Unit appointed CMP to serve as a coordinating agency to resolve issues related to consolidation. Beyond policy support, however, evidence suggests that institutional support for the School Consolidation Policy was more limited in scope than that provided to the PPP Node. According to RSU, CMP did not provide direct technical assistance to RSU staff. CMP was not formally linked with RSU. Communication channels were open, however. CMP correspondence with RSU indicates that CMP called upon RSU support in conducting school consolidation workshops at the district level. RSU reported providing CMP with guidance on the School Consolidation Policy and receiving invitations to CMP events. According to RSU, interaction between CMP and RSU was otherwise administered “on a needs basis.” 22 RSU touted the potential for coordination between RSU and CMP, given their common purpose, but also recognized constraints to coordination, given that CMP was active only in CMP communities. District- and School-Level Support for Education Sector Reforms At the implementation level, although CMP’s mandate principally applied to the pre-transition phase for outsourcing education management to the private sector, EMO staff recognized the utility of CMP’s support for community engagement and mobilization activities in EMO schools. However, lack of clarity among EMO school managers regarding stakeholder roles and responsibilities suggested greater need for post-transition support for EMO management. With respect to consolidation, CMP provided support at the district and school levels mainly through orientation workshops at the district level and through regular community engagement and capacity-building intervention at the school level. Improved school facilities and pre-existing community perceptions of EMOs were the most influential factors affecting enrollment decisions and community support for EMO management. Complementary CMP-supported community mobilization activities designed specifically to support the transition to EMO management allayed concerns about the transition among teachers and management staff. CMP community engagement and mobilization efforts appear to have continued unabated in EMO schools following the EMO transition. Each of the three EMO school coordinators interviewed for the evaluation affirmed CMP’s role in coordinating community engagement events. Two specifically reported health and hygiene activities. Two of the three EMO school coordinators also cited the value of CMP’s efforts to increase enrollment (the third coordinator explained that the school had exceeded its enrollment capacity), despite the fact that one of those schools designated its own “social mobilizer.” 22 RSU’s lack of knowledge of CMP’s support for the School Consolidation Policy at the provincial level is possibly due to staff turnover. The School Consolidation Policy was passed in 2012 and amended in 2015. At the time of the evaluation, implementation of the policy had advanced significantly. According to RSU, GoS had merged 3,500 schools and established 1,350 campus schools. Sindh Community Mobilization Program – Midterm Performance Evaluation 27 Evidence suggests, however, that improved facilities associated with new construction and positive reputations of EMOs had a pronounced effect on enrollment in and support for EMO schools. Except for communities’ initial concerns that EMO schools would charge fees, the evaluation team did not find significant community resistance to the EMO transition. On the contrary, parents from each of the five EMO schools in the sample were highly supportive of the changes, citing improved quality of education, improved facilities, EMO reputations, and improved security. The transfer of students from non-EMO to EMO schools further illustrated the degree of community acceptance of, or support for, the EMO model. Parents from four of the five EMO schools in the sample noted that the EMO’s reputation had positively influenced their decision to enroll their children in that school. 23 One parent of a child enrolled in an EMO school in Khairpur was explicit, noting, “Our children came here from private schools, and the reason why we enrolled them here was IBA.” The head teacher in that same school further commented that most of his students had transferred from private or other government schools. CMP efforts to sensitize communities to the EMO model likely affected these sentiments, but the evidence also shows a community proclivity to support EMO management. EMOs did, however, encounter resistance, at least at initial stages in the transition process, from teachers and incumbent management staff within the schools. EMO coordinators from two schools in Khairpur, for example, described anxieties among teachers, head teachers, SMCs, and education sector officials, as teachers had to adjust to new management systems and the arrival of additional teaching staff, head teachers had to relinquish authority during consolidation, and SMCs and district officials in some cases interpreted the transition as an indication of their own shortcomings. One SMC member in Khairpur described his reservations, which later proved unfounded, noting: “When we heard the EMO was coming, we were disturbed about whether the EMO would terminate all the teachers and take over the school. What will happen, will the EMO let us work or not? Later we learned that they will hire more teachers and provide good education to our children.” Likewise, a high-ranking provincial education official expressed his reservations and dismay that school management was being outsourced: “I have reservations. Why have these newly constructed schools been given to EMOs? Those newly constructed schools are run by the same government school head teachers. When those schools were in bad condition, without any facilities, government teachers were running those schools. All those schools have the same employees, even the support staff is the same. At that time our teachers were running those schools with devotion. So why can’t they run those schools now? Yet we accept that EMOs are running the schools with standards, while we are not up to that standard.” According to IRD, CMP implemented community mobilization efforts specifically designed to allay concerns about the transition to EMO management and garner stakeholder support for EMO reforms. Head teachers and SMC members in at least three EMO schools recognized CMP’s efforts. A head teacher in Khairpur noted: “All teachers participated in [the] trainings and visited [other EMO] schools. CMP informed us about EMOs and explained [the need for cooperation]. EMOs have their own rules and regulations. We were worried before … What will happen, will they terminate us?... Then CMP members encouraged us [and explained that] we will work in coordination, that EMOs will have a separate set-up, and that they will support us to improve the quality of education. We benefitted from this information.” 23 Parents describing their decision to enroll children in EMO schools may have been referring to the decision to transfer their children from a non-EMO public or private school to an EMO school; the decision to enroll first-time enrollees, i.e., children entering first grade; or, in a few cases, the decision to enroll previously out-of-school children in an EMO school. In other words, this decision was a matter of school choice as much as a matter of new enrollment. Sindh Community Mobilization Program – Midterm Performance Evaluation 28 Lack of clarity regarding stakeholder roles and responsibilities precluded effective stakeholder coordination following the transition to EMO school management. District support teams were a useful mechanism for improving stakeholder coordination. EMO staff indicated a lack of clarity regarding stakeholder roles and responsibilities in the post-transition phase, creating some confusion in the conduct of post-transition school management activities. This lack of clarity may have been due to a lack of implementation guidance from the GoS, a lack of EMO awareness of available guidance (including in the EMOs’ own concession agreements), or lack of EMO familiarity with government processes. According to one EMO respondent: “The schools were handed over to us and it was announced that now this is a high school. Considering we are a private organization, we didn’t know how the government worked and what their mechanisms were. We were not given any guidelines to follow in terms of how we are supposed to manage [the school] and what criteria we must meet.” The EMO respondents viewed CMP as being in close association with the government on issues pertaining to management of the transition, but felt that association had not translated into a defined support role for CMP inside the school. An EMO official noted: “CMP’s primary role is before the transition to the EMO. They work initially with SMCs and other ground-level activities that do not link up with EMOs… We don’t have any specific document where CMP roles and responsibilities are defined. They are only supporting the government, as their primary role is on the government side, for example with RFPs, notification of government bidding, and review of RFPs … But we feel no support from their side.” This lack of clarity regarding CMP’s role led EMOs to turn to CMP for support that lay beyond CMP’s mandate and budgetary capacity. For example, an EMO coordinator in Khairpur, having expressed some confusion about whether CMP was a government representative or a mediator, noted his disappointment following unsuccessful requests for CMP’s assistance in transferring teachers. The lack of clarity also led to concerns of duplication of effort. This same EMO coordinator noted that coordination with CMP improved after “we told them we both have to work for the same activity at the same place, so instead of wasting double energy and time, let’s plan things together and do it.” To improve coordination and communication among stakeholders, CMP sponsored the DSTs and, in at least one community, a taluka management committee. District managers for both EMOs cited DSTs as useful forums for communication with both CMP and DEOs. Nonetheless, these forums would be more effective if stakeholders had clarity about their roles and responsibilities. For example, one EMO manager noted, “Honestly speaking, the role of CMP is not clearly defined so that we can highlight this in the DST forum… CMP’s role is not mentioned so on what basis can we blame them?” CMP facilitated implementation of the Consolidation Policy through regular community engagement activities and orientation of education managers and head teachers. In the RSU’s view, the appropriate way for CMP field staff to provide consolidation guidance to CMP schools was to frame it within the scope of CMP’s regular community engagement activities. As an RSU official explained, “Since CMP is actively working within [those communities], I’m sure they would be more effective than the reform unit because they have been in coordination with the communities at the local level. [CMP has] built the trust of [those communities].” Evidence suggests that CMP support eased the consolidation process at the district level. DEOs and other education officials, for example, were perceptive to the challenges of consolidation, as well as the value of CMP support. One of the three DEOs interviewed, as well as a provincial education official, reported receiving technical guidance from Sindh Community Mobilization Program – Midterm Performance Evaluation 29 CMP on consolidation processes. Another DEO cited the value of CMP supply-side activities and coordination support at the school level, noting: “The consolidation policy is difficult to implement in big schools … CMP has supported consolidated schools by providing them small grant schemes and developing the [school improvement plan]. For example, CMP provided a waterline in Khair Pur Juso school. They have done a good monitoring job. Meetings are carried out through CMP, in which the community also gets involved.” Two head teachers of CMP schools (of 10 CMP schools that had undergone consolidation) reported having received training on consolidation procedures and processes. Consolidation, however, seemed to pose few management challenges at the school level. Head teachers and SMC members from only 3 of the 14 CMP schools that underwent consolidation cited challenges associated with the consolidation process. These were administrative in nature—one head teacher, for example, cited difficulties monitoring and approving leave of teachers at merged schools. Parents in only 1 of the 14 communities expressed any apprehension about consolidation, and that apprehension was alleviated by the fact that after the consolidation, their children could attend secondary school in the same building as primary school. Conclusion The RSU initiated the school consolidation process in 2012 and so at the time of the midterm evaluation was well-positioned to manage that process. At the school level, consolidation was often only an administrative task. Thus the evidence did not indicate a strong need for CMP institutional support or technical inputs for this set of reforms at either the policy or implementation level. The story was different with respect to CMP’s support for the transition to private sector school management. Given resource constraints at the PPP Node, CMP’s support for the node was indispensable. At the school level, however, there was a lack of clarity regarding stakeholder roles and responsibilities and CMP’s post-transition role in the implementation of the EMO model, often to the frustration of school and EMO staff. Such growing pains will likely diminish over time. EMOs recognized CMP’s contributions to community engagement and expressed their desire for continued CMP support in the post-transition phase. Should CMP continue its efforts to build and institutionalize the capacity of the PPP Node, while strengthening stakeholder collaboration in the initial stages of the post-transition phase, EMOs may emerge as a viable mechanism for continuing CMP engagement and mobilization practices following CMP’s conclusion. An EMO coordinator described an ideal scenario by which EMO schools could gradually phase out CMP: “[CMP’s] visits are reduced with the passage of time. They are more confident that EMO can run the school. CMP provided good support in mobilizing the parents and provided us the lists of out-of-school children. They made SMC functional, and SMC members are cooperating with us.” Question 4: Relevance of CMP Performance Indicators and Targets Are CMP indicators appropriately defined, to what extent has CMP achieved its targets, and are these targets relevant to achieving program objectives? Question 4 assesses the adequacy of enrollment as a goal-level indicator of access to schooling and the relevance of enrollment targets to the achievement of program objectives.24 CMP reports on two goal￾level indicators pertaining to enrollment. Of interest to the evaluation team is the second, or “Goal-b,” 24 The evaluation team elected to assess enrollment as an indicator in consultation with USAID. Other performance indicators in CMP’s Monitoring, Evaluation, and Learning plan were not considered for this purpose. Sindh Community Mobilization Program – Midterm Performance Evaluation 30 indicator, which measures the number of U.S. Government (USG)-supported out-of-school children (boys and girls) who begin studying in targeted schools. 25 The evaluation team assessed the appropriateness of the enrollment indicator based on whether it is meaningful (e.g., valid, sufficient, and well-measured), practical (accounting for availability of data and feasibility of alternate indicators), and attributable. The team then assessed the relevance of CMP’s enrollment targets based on whether those targets can be achieved within the project time frame and whether they contribute to the achievement of program objectives. Evaluation findings indicate that aggregate enrollment is insufficient and potentially misleading as an indicator of CMP’s effectiveness in increasing access to schooling. The enrollment indicator, as currently reported, is likely to overstate overall access to schooling as well as CMP’s contributions to access in CMP schools, diminishing the relevance of associated enrollment targets. Enrollment as a Meaningful Indicator of Access to Education Retention and attendance are complementary indicators of access to education not sufficiently measured by CMP.26 The enrollment indicator as currently measured does not capture a child’s physical presence in the classroom or students’ retention at higher levels of education. In other words, the enrollment indicator does not distinguish what age groups are gaining greater access to schooling or the extent to which access has increased on a day-to-day or year-over-year basis. The fact that no reliable and easily obtainable attendance data exists limits the practicality of measuring attendance on a regular basis. However, the distinction is important, as the factors that affect enrollment, attendance, and retention are different (see Question 1). Whereas parents identified the top factors affecting enrollment to be their awareness of the importance of education and their motivation to educate their children, the top barrier to retention was distance between home and school and associated transportation costs. Attendance, on the other hand, is frequently contingent upon preferences for a child to contribute to household income or chores (including seasonal labor). As the barriers to access change, so should the nature of interventions to overcome those barriers. An exclusive focus on one access indicator—enrollment—can affect achievement of other indicators. In CMP schools where resources are strained or school facilities are inadequate to accommodate a growing student body, for example, increasing enrollment without addressing supply-side constraints may deter students from attending and thus limit CMP’s effect on attendance. A parent in Karachi, for example, noted: “In two rooms and one veranda three schools are running … The room is small and there are a number of children sitting in one room; some of them are sitting on the floor. This is the big challenge. Our children want to go to school but because of space we face problems.” It follows that the focus on enrollment as the primary indicator of program performance and access to schooling can overshadow CMP supply-side interventions that increase attendance. For example, an SMC member in Karachi attributed more regular attendance to school improvements funded by SMC 25 CMP measures two goal-level indicators pertaining to access to schooling. The “Goal-a” performance indicator measures “number of learners enrolled in primary, elementary and secondary USG assisted schools or equivalent USG assisted non-school based settings.” CMP calculates this figure by adding up all students enrolled in CMP-supported schools. The “Goal-b” indicator measures “number of out-of-school children newly enrolled or re-entered in the education system with USG assistance.” CMP defines out-of-school children as children between the ages of 5 and 18 who are not currently enrolled in the mandatory course of study in either a formal or informal public or private setting that offers education ultimately culminating in a diploma accredited by the Government of Pakistan. Re-enrollment refers to children who were previously enrolled in such an education setting but discontinued study and have since become re-enrolled with USG assistance. A third target group for Goal-b is newly enrolled students who never previously enrolled to study in such an education setting but have become enrolled through USG assistance. 26 CMP periodically measured attendance through head counting. This was helpful in improving attendance by sensitizing teachers and students, but was not reported on as a key performance indicator. Sindh Community Mobilization Program – Midterm Performance Evaluation 31 small grants, noting, “Due to [the] small grant scheme, the condition of the school improved. The parents who were reluctant to send their children regularly due to lack of electrical fans and water are now satisfied with school conditions.” Where schools were fully enrolled, enrollment estimates did not effectively capture the full achievement of CMP objectives. Thirteen head teachers reported insufficient space or inadequate facilities to accommodate additional enrollment. Head teachers had refused additional enrollment in at least three sample schools. Even CMP reported shifting its operational focus from fully or overenrolled schools to schools with available capacity. This did not mean, however, that all enrolled students in fully or overenrolled schools were in attendance. For example, the head teacher in Khoro Goth described a scenario by which enrollment far exceeded school capacity in a newly constructed and upgraded building and how the school responded: “The enrollment has been increased because of CMP interventions. We have 14 classrooms in our school and now we don’t have the space to enroll more children. The school shifted to this newly constructed building almost a year ago. Last year when the school started in the new building, the enrollment was only 246, now that has increased to 642 … We have only 14 classrooms and each class has around 50 students. The class space is for a maximum of 40–45 students. The classrooms have become congested for students. We don’t have extra classes where we can accommodate children … We cannot enroll kindergarten students because classes are fully packed … We have informed parents we are not taking enrollments for grades 4–5 and they can come back next year in April.” In such cases, attendance or enrollment/classroom ratio may serve as better indicators of the effectiveness of those CMP activities that improve school management and facilitate other supply-side improvements without the express goal of increasing enrollment. As currently measured, enrollment estimates cannot isolate attribution of enrollment to CMP and are thus likely to misrepresent CMP’s contribution to increased access. The school year begins in April, and CMP conducts enrollment drives in May. CMP field staff measures Goal-b enrollment as all children enrolled after May. This method, while practical, cannot isolate the contribution of CMP engagement and mobilization activities from the effect of external factors or other interventions, particularly school rehabilitation or construction. Given the degree to which school facilities and amenities affect parents’ enrollment decisions, this enrollment measure is likely to overestimate the effect of community engagement and mobilization on access to schooling. As one district official put it, “Enrollment is not an issue. Where there is a school in the village, students are available. You will find many schools where there is no further [space] for children to be enrolled.” Achievement of Enrollment Targets Reported enrollment figures meet CMP’s targets, but likely overstate the effect of community engagement and mobilization on enrollment. As of September 30, 2016, CMP reported having enrolled 38,812 additional students, exceeding its target of 25,000. As described in Question 1, however, the evaluation team calculated alternative estimates of CMP’s direct contribution to enrollment using statistical models to compare CMP schools with a comparable group of non-CMP schools, controlling for potential confounding factors.27 The 27 The shortcomings of this approach are the following: (a) The implementing partner cannot perform such estimations at the school level on a monthly and quarterly basis; (b) the estimation itself remains vulnerable to shortcomings in school census data; and (c) while known confounding factors can be controlled for, a number of factors remain unknown which may differentially affect enrollment in the treatment and comparison groups. Sindh Community Mobilization Program – Midterm Performance Evaluation 32 evaluation found a total increase in enrollment of 7,882 students, including 4,020 girls and 3,783 boys, attributable to CMP activities (see Annex 9). The significant disparity between these estimates of CMP’s effect on enrollment and the increases reported by CMP suggests that established targets exaggerate expectations of enrollment growth resulting from community engagement and mobilization. Relevance of Enrollment Targets The relevance of CMP’s enrollment targets diminishes as (a) the definition of access to schooling expands and (b) the full breadth of CMP activities is considered. Regardless of how they are determined, enrollment targets as a performance goal are problematic. Enrollment targets for CMP schools do not consider, for example, transfers of students from non-CMP schools to CMP schools, nor do they consider students who enroll to become eligible for stipends or schools that exaggerate enrollment to become eligible for government funds. Nor do year-over-year targets capture the longer-term positive impact of CMP’s activities on the socialization of education. As discussed, evidence from parent interviews demonstrates that heightened awareness of the value of education and motivation to educate one’s children were key factors contributing to enrollment. Parents in 11 communities reported increased community recognition of the value of education, particularly for girls. These changes take time and cannot be fully captured during CMP’s period of performance. More importantly, enrollment targets do not fully capture many of the engagement and capacity-building activities discussed in this report, including activities to strengthen SMCs, which engage parents in school processes beyond the enrollment decision, and activities that engage DEOs, which have supply-side implications that affect attendance and retention as much as—or more than—they affect enrollment. Thus, as the conception of access expands to consider activities that address barriers specific to attendance and retention, the relevance of enrollment targets diminishes. Conclusion Aggregate enrollment has limited meaning as a stand-alone indicator of access to schooling. Enrollment fails to account for key aspects comprising access to education, namely continued enrollment (i.e., retention) and attendance. Barriers to retention and attendance differ from barriers to enrollment, as do strategies for overcoming these barriers. Moreover, enrollment does not capture the full range of outcomes of CMP activities. A more meaningful set of indicators to measure access to education must account not just for barriers to enrollment, but also barriers to attendance and retention. Indicator targets, moreover, should consider schools’ capacity to accommodate a greater number of students in the classroom. Not only might a more comprehensive construct of access provide more meaningful estimates of CMP’s performance, but more practical and context-specific targets may spur supply-side interventions, from SBEP or others, where needed. CONCLUSIONS The evaluation validates a strong link between community engagement and increased enrollment in CMP target communities. CMP enrollment campaigns and community events have raised awareness of education processes and socialized the notion that education is valuable and accessible, and have done so within a dynamic web of compelling social, economic, and supply-side factors affecting parents’ decisions about their children’s enrollment. The availability of quality schooling within reasonable proximity to the home remains a particularly decisive factor influencing parents’ enrollment decisions and one which gains increasing importance as children, particularly girls, grow older. It is thus significant that CMP has extended its focus on access beyond strengthening parents’ desire and ability to enroll Sindh Community Mobilization Program – Midterm Performance Evaluation 33 their children in schools, to empowering parents to be involved in education processes to ensure their children’s regular and continued attendance. In this regard, the evaluation confirms CMP’s role as an integral component of the larger SBEP. CMP’s support for SBEP’s supply-side interventions resides not only in enrolling children in newly constructed schools, but in mobilizing communities and education officials around school management changes and influencing how parents and children engage with the school after children have enrolled. CMP’s approach to community engagement and mobilization has centered around the SMC. The evaluation finds that the SMC, as a formally recognized institution with budgetary responsibility and an inclusive membership, is an appropriate vehicle for community engagement. CMP has made great strides in activating SMCs, strengthening SMC processes, and broadening the scope of SMC activities through the formation of SMC subcommittees. Interaction between SMCs and parents has increased in CMP communities. But for SMCs to be truly inclusive requires increasing communities’ confidence to interact with SMCs and communities’ understanding of SMC roles and responsibilities. Also, in many CMP communities, SMCs are still predominantly staffed by men. This dynamic may be shifting with the advent of SMC subcommittees, but nonetheless precludes effective SMC engagement with women. CMP’s contribution to SBEP further derives from CMP’s convening authority. CMP’s ability to draw education officials to community events has created opportunities for community interaction and resource mobilization. CMP’s leadership in stakeholder engagement has also emerged in its support for education sector reforms, principally through the launch of district-level forums for information sharing and resolution of issues pertaining to the outsourcing of school management to the private sector. Nonetheless, the effectiveness of CMP’s provincial and school-level support for PPPs in the education sector and school consolidation has been mixed. On the one hand, CMP has successfully facilitated a pilot of the EMO model, supporting the GoS’s PPP Node by developing operating procedures; assisting the transition process, from RFP to handover of schools to selected EMOs; and sponsoring district stakeholder forums. On the other hand, a lack of clarity on stakeholder roles and responsibilities following the transition to EMO management persists around the EMO model. While post-transition technical assistance for EMO school management lies beyond the scope of CMP as CMP is currently constituted, stakeholder feedback suggests there are technical shortcomings in the implementation of the EMO model that can be amended by future USAID programming. Where consolidation is concerned, well-established practices of the RSU preclude the need for regular CMP support at the provincial level. Across both sets of reforms, however, CMP engagement and mobilization efforts and support for school improvement processes have helped to ease communities’ adjustment to change. Finally, recognition of the relationship between community engagement and access to schooling was cause for review of CMP program indicators related to access. CMP’s chief access indicator measures enrollment. The evaluation concluded that reported enrollment data overestimates CMP’s contribution to enrollment. However, the evaluation also concluded that enrollment is an imperfect indicator of access, given supply-side limitations to enrollment, barriers to attendance and retention, and longer￾term effects of CMP’s interventions. Such limitations in turn diminish the relevance of CMP’s enrollment targets. With proper resources, the prospects for sustaining CMP’s interventions appear strong. CMP engagement and mobilization activities have worked through established institutional partners—i.e., SMCs—that enjoy government support and ownership. In the case of CMP support for educational reforms, the RSU is moving the consolidation process forward, and once processes, roles, and responsibilities for private sector school management have been institutionalized, EMOs represent a logical partner for continued community engagement. Nevertheless, the effectiveness of CMP Sindh Community Mobilization Program – Midterm Performance Evaluation 34 engagement activities will continue to depend on the ability of the GoS, SBEP, and other stakeholders to reduce supply-side constraints. RECOMMENDATIONS Future education programming should include attendance and retention as indicators for access to schooling. Targets for access indicators should account for supply-side (i.e., school capacity) and demand-side (i.e., number of out-of-school children) factors over time. Enrollment is a necessary but insufficient indicator for measuring access to schooling. Nor is it possible to measure change in enrollment attributable to CMP at the school level without data from a proper comparison group. However, indicators that consider attendance and retention can more adequately capture a program’s effectiveness in improving day-to-day and year-over-year changes in access to schooling. The remaining life of CMP, current scope of CMP programming, and available resources may limit the feasibility of shifting CMP’s focus from enrollment to attendance and retention. For future programs, however, an appropriate set of indicators should, to the extent practical, aim to accomplish the following: 1. Provide a meaningful measure of attendance at multiple junctures during the school year. Future programming may use attendance registers to monitor attendance, but this data should be independently verified. Conducting head counts at strategic intervals, for example, could account for variance in attendance caused by migration or other seasonal factors. Depending on program objectives, attendance data can be reported and targets set in a number of ways, including as a percentage of students in attendance for a benchmark number of days or average percentage of daily attendance over a given period of time. 2. Provide a meaningful measure of retention. Future programs may provide a meaningful measure of retention by monitoring general register data by class and comparing enrollment across years. In this way, programs can report retention as an estimated percentage of students advancing to the next grade. However, by treating class-level enrollment data in the aggregate, these estimates would not capture new enrollment or reenrollment of students who have dropped out. To better understand the effect of community engagement on new enrollment, dropout rates, and retention, USAID should consider sentinel indicators that monitor supply-side (i.e., school capacity) and demand-side (i.e., out-of-school children) factors over time at the community level in target communities, beginning with a baseline estimate for each sentinel indicator prior to program implementation. Enrollment targets should reflect baseline assessments of these indicators and can be modified during the life of future programs to reflect learning and changes in the local context. CMP should have a strategic plan to mobilize or advocate for stopgap supply-side interventions and, where appropriate and practical, shift to attendance and retention activities in overcrowded schools. Schools’ capacity in target districts is constrained by insufficient space and number of teachers. Schools overenroll with the anticipation that not all students will attend, while crowded classrooms and high student-teacher ratios in turn compromise attendance and retention. CMP should utilize district and provincial forums such as DSTs and steering committees to further advocate with relevant government departments for addressing supply-side constraints in schools. Stopgap supply-side interventions could include, for example, arranging alternative or temporary classroom space for overcrowded schools or Sindh Community Mobilization Program – Midterm Performance Evaluation 35 finding alternative spaces during construction. Meanwhile, once enrollment targets have been achieved, CMP should consider strategies to broaden its focus from enrollment to attendance and retention activities. Enrollment events, general body and subcommittee meetings, and other SMC activities in overcrowded schools should be equipped to shift their focus to attendance. CMP activities to strengthen SMC processes for engagement and school improvement should include an explicit knowledge-building component for SMC general bodies. When parents understand SMC processes, roles, and responsibilities, they are more likely to participate in community engagement and school improvement activities and more likely to hold school management accountable. CMP currently engages with SMC general bodies through general meetings and SMC subcommittees. Additional knowledge-building exercises for members of SMC general bodies could encourage greater participation of parents, especially women, in SMC and subcommittee activities. CMP should arrange for formal transfer of knowledge and best practices for EMO management to the PPP Node and DSTs. CMP successfully piloted the EMO model and facilitated the transition of four schools to EMO management. CMP should assist the PPP Node and DSTs in establishing a process for institutionalizing standard operating procedures and best practices for EMO management. A short-term consultant could be commissioned for this purpose. Future programming could include similar technical assistance for the post-transition phase when outsourcing school management to the private sector. CMP should put in place a plan to institutionalize stakeholder coordination at the district level to support EMOs before and after the transition to EMO school management. CMP’s support for the EMO model has been pronounced in the pre-transition phase at the provincial level. Although longer-term support for EMOs in the post-transition phase is beyond CMP’s scope, there is still a need for greater support for stakeholder coordination and clarification of stakeholder roles and responsibilities at the district and school levels in the initial stages of the post-transition phase. CMP should continue to assist the GoS in strengthening and institutionalizing coordination mechanisms, such as the DST forums, for district administration officials, DEOs, and EMOs, and support an exit strategy for handing over responsibility for convening such forums. Such activities may require an expansion of CMP’s scope. CMP should continue its district- and school level support to RSU for implementation of the School Consolidation Policy. To fill implementation gaps, CMP could consider an exercise to assess needs and identify lessons learned from consolidated schools to facilitate RSU in improving implementation of the School Consolidation Policy. CMP support should be need-based. CMP’s experience working at the school level may offer an opportunity to assess successes and challenges in consolidated schools, document lessons learned, and report back to RSU with recommendations for improving implementation mechanisms. Needs assessments may address issues of feasibility, technical assistance needs of DEOs, and capacity-building needs of head teachers. CMP can coordinate with RSU to address gaps and implementation challenges. Sindh Community Mobilization Program – Midterm Performance Evaluation 36 ANNEXES Annex 1: Statement of Work Sindh Community Mobilization Program – Midterm Performance Evaluation 37 Sindh Community Mobilization Program – Midterm Performance Evaluation 38 Sindh Community Mobilization Program – Midterm Performance Evaluation 39 Sindh Community Mobilization Program – Midterm Performance Evaluation 40 Sindh Community Mobilization Program – Midterm Performance Evaluation 41 Sindh Community Mobilization Program – Midterm Performance Evaluation 42 Annex 2: Assignment Work Plan Sindh Community Mobilization Program – Midterm Performance Evaluation 43 Sindh Community Mobilization Program – Midterm Performance Evaluation 44 Sindh Community Mobilization Program – Midterm Performance Evaluation 45 Sindh Community Mobilization Program – Midterm Performance Evaluation 46 Sindh Community Mobilization Program – Midterm Performance Evaluation 47 Sindh Community Mobilization Program – Midterm Performance Evaluation 48 Sindh Community Mobilization Program – Midterm Performance Evaluation 49 Sindh Community Mobilization Program – Midterm Performance Evaluation 50 Sindh Community Mobilization Program – Midterm Performance Evaluation 51 Sindh Community Mobilization Program – Midterm Performance Evaluation 52 Sindh Community Mobilization Program – Midterm Performance Evaluation 53 Sindh Community Mobilization Program – Midterm Performance Evaluation 54 Sindh Community Mobilization Program – Midterm Performance Evaluation 55 Sindh Community Mobilization Program – Midterm Performance Evaluation 56 Sindh Community Mobilization Program – Midterm Performance Evaluation 57 Sindh Community Mobilization Program – Midterm Performance Evaluation 58 Sindh Community Mobilization Program – Midterm Performance Evaluation 59 Sindh Community Mobilization Program – Midterm Performance Evaluation 60 Sindh Community Mobilization Program – Midterm Performance Evaluation 61 Sindh Community Mobilization Program – Midterm Performance Evaluation 62 Sindh Community Mobilization Program – Midterm Performance Evaluation 63 Sindh Community Mobilization Program – Midterm Performance Evaluation 64 Sindh Community Mobilization Program – Midterm Performance Evaluation 65 Annex 3: Team Composition and Conflict of Interest Disclosures Team composition and conflict of interest disclosures were removed to preserve the confidentiality of evaluation team members. They are available from PERFORM upon request. Sindh Community Mobilization Program – Midterm Performance Evaluation 66 Annex 4: Interviews Conducted by the Evaluation Team Table 2 details key informant interviews conducted by the evaluation team, including 24 school-level interviews with head teachers. TABLE 2: KEY INFORMANT SELECTION Administrative Level Respondent Type Number of Interviews Province Office of the Director, School Education and Literacy Department 1 Reform Support Unit 1 Project Management Unit 1 Public Private Partnership Node 1 IRD Karachi 1 District District/taluka education officers 5 IP/sub IP district managers 8 a School Head teachers 24 Grand Total 42 a. The sample of implementing partners includes two regional and two district officials of IRD, one regional and one district official of IRC, and two district officials of HANDS. This sample captures at least one official in each district except Jacobabad and Karachi. Sindh Community Mobilization Program – Midterm Performance Evaluation 67 Annex 5: Data Collection Instruments GROUP INTERVIEW DISCUSSION GUIDE (PARENTS) BENEFICIARY SCHOOLS Respondent Data GI(P)1. Facilitator name: _________________ GI(P)2. Facilitator ID: ___________________ GI(P)3. Date: _________________________ GI(P)4. Respondent name: _______________ GI(P)5. School:________________________ GI(P)6. Village: ________________________ GI(P)7. Union Council: __________________ GI(P)8. District: _______________________ GI(P)9. Intervention type: (circle one - can be multiple) a. Community engagement/mobilization b. School construction/rehabilitation c. School consolidation d. Public-private partnership/ EMO e. Small grant/neighboring GI(P)10. Respondent (respondent information regarding number of children, age, gender will be gathered during screening process. Respondent 1 Respondent 5 Respondent 2 Respondent 6 Respondent 3 Respondent 7 Respondent 4 Respondent 8 GI(P)11. Starting time: ________________________ GI(P)12. End time: ___________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the effectiveness of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically, we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. Sindh Community Mobilization Program – Midterm Performance Evaluation 68 GI(P)13. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)14. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION A: RESPONDENT BACKGROUND (ASK ALL RESPONDENTS) GI(P)15. Gender [OBSERVE and CIRCLE ONE NUMBER] 1 Male 2 Female GI(P)15. What is your age (in years)?_________________________ [If respondent doesn’t know his or her age, ask for his/her year of birth. Calculate the age and enter above.] GI(P)16. How long have you been living in this village (in years)? _________________ [If respondent doesn’t know how many years, ask which year respondent arrived and calculate the number of years.] GI(P)17. What is your marital status? [CIRCLE ONE NUMBER] 1 Never married 2 Married 3 Divorced 4 Widowed 5 Refused to answer Sindh Community Mobilization Program – Midterm Performance Evaluation 69 GI(P)18. What is the highest level of education you have completed? [CIRCLE ONE NUMBER FOR EACH RESPONDENT] Level of educational attainment Respondent 1 2 3 4 5 6 7 8 No education 1 1 1 1 1 1 1 1 Primary 2 2 2 2 2 2 2 2 Middle 3 3 3 3 3 3 3 3 Matric 4 4 4 4 4 4 4 4 F.A/FSc. 5 5 5 5 5 5 5 5 Technical education (diploma) 6 6 6 6 6 6 6 6 BA/BSc 7 7 7 7 7 7 7 7 MA/ MSc. or above 8 8 8 8 8 8 8 8 Professional degree (e.g. medical degree like M.B.B.S, lawyer, etc.) 9 9 9 9 9 9 9 9 Madrassa education 10 10 10 10 10 10 10 10 Other (e.g. religious education) 11 11 11 11 11 11 11 11 If other, please specify: 12 12 12 12 12 12 12 12 Refused to answer 13 13 13 13 13 13 13 13 GI(P)19. What is your occupation? Respondent Response 1 2 3 4 5 6 7 8 Sindh Community Mobilization Program – Midterm Performance Evaluation 70 SECTION B: FACTORS AFFECTING ENROLLMENT Factors Affecting Enrollment GI(P)20. What are the primary factors you considered when you decided whether to enroll your child/children in school? Did your decision depend on the child/children’s age—how and why? Did your decision depend on their sex—how and why? [Probe how/why.] [Probe for external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] Respondent Response 1 2 3 4 5 6 7 8 GI(P)21. If your child is enrolled in this school, why did you choose to enroll your child in this school? [Probe for cost, distance, access, better education, school improvements, increased awareness about the school, parents’ participation, etc.] Respondent Response 1 2 3 4 5 6 7 8 GI(P)22. If you have other children of age to attend this school who are not enrolled, why are they not enrolled in this school? Are they attending another school or learning institution? What kinds of institutions? Why did you choose that institution? [Probe for gender; cost, distance, better/different type of education, social pressure, beliefs, etc.] Respondent Response 1 2 3 4 5 6 7 8 Sindh Community Mobilization Program – Midterm Performance Evaluation 71 GI(P)23. If you have children you chose not to enroll or keep in school, what are the barriers or challenges, if any, that affected your decision not to enroll your child/children in school? Did your decision depend on the child/children’s age—how and why? Did your decision depend on their sex—how and why? [Probe how/why.] [Probe for external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] Respondent Response 1 2 3 4 5 6 7 8 Factors Affecting Retention/Attendance GI(P)24. What are (were) the primary factors you consider(ed) when you decide(d) whether to keep your child/children in school? Does (did) your decision depend on the child/children’s age—how and why? Does (did) your decision depend on their sex—how and why? [Probe how/why] [Probe for external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness)] Respondent Response 1 2 3 4 5 6 7 8 GI(P)25. If you chose not to keep a child/children in school, what, if any, barriers did you face to keeping your child/children in school? What are (were) they? Are (were) these barriers different for different age groups? Are (were) they different for boys and girls? [Probe for drop-out and never enrolled, what, how/why] [Probe for external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness)] Respondent Response 1 2 3 4 5 6 7 8 Sindh Community Mobilization Program – Midterm Performance Evaluation 72 SECTION C: EFFECT OF CMP ON ENROLLMENT GI(P)26. Have any activities in your community in the past 2 years affected your decisions about enrolling your child/children in school or keeping them in school? Explain how and why, provide examples? [Probe for specific CMP activities and how or why not for differential effects for different types of children, including gender and age.] Respondent Response 1 2 3 4 5 6 7 8 SECTION D: EFFECTIVENESS OF CMP COMMUNITY ENGAGEMENT ACTIVITIES GI(P)27. Have you interacted, or do you interact, with your child’s school or the school management committee (SMC) of your child/children’s school in any way? If yes, how? [Probe for participating in the SIP/SMP process, receiving information from the SMC, communicating needs to the SMC, etc.).] GI(P)28. Have you discussed matters related to your child’s school with SMC members outside formal SMC meetings? Please explain. [Probe for specific examples: when, where, for what purpose] Respondent Response (Yes/No) 1 2 3 4 5 6 7 8 GI(P)29. Has your interaction with your child’s school or the SMC of your child’s school changed during the past two years? If yes, how and why? [Probe for participation in the SIP/SMP process, receiving information from the SMC, communicating needs to the SMC, etc.. Why have these things changed?] GI(P)30. Has your interaction with your child/children’s school or the SMC of your child/children’s school been effective or ineffective for resolving issues related to your child/children’s enrollment or retention? Has the school’s or the SMC’s responsiveness to these issues increased, decreased, or stayed the same in the past two years? How and why? Give examples. [Probe for direct and indirect effects.] Sindh Community Mobilization Program – Midterm Performance Evaluation 73 SECTION E: CONSOLIDATION (Ask only for consolidated schools) GI(P)31. Did you support consolidation of your school? Why or why not? Has your support changed over the past two years? If yes, how and why? GI(P)32. Has consolidation affected your interaction with the SMC? Explain how and why. Give examples. GI(P)33. Has consolidation affected your interaction with the school? Explain how and why. Give examples. [If necessary, probe school representatives, including child’s teacher, head teacher, other.] GI(P)34. Has consolidation affected your decision to enroll your child/children in school or keep them in school? If yes, explain how and why. [Probe for differences in decisions by sex, age, and the interaction of the two] SECTION F: PUBLIC-PRIVATE PARTNERSHIP (Ask only for EMO schools) GI(P)35. Did you support the transition to EMO school management in your school? Why or why not? Has your support changed over the past two years? If yes, how and why? GI(P)36. Has the transition to EMO management affected your interaction with the SMC? Explain how and why. Give examples. GI(P)37. Has the transition to EMO management affected your interaction with the school? Explain how and why. Give examples. [If necessary, probe school representatives, including child’s teacher, head teacher, other.] GI(P)38. Has the transition to EMO management affected your decision to enroll your child/children in school or keep them in school? If yes, explain how and why. [Probe for differences in decisions by sex, age, and the interaction of the two.] Sindh Community Mobilization Program – Midterm Performance Evaluation 74 GROUP INTERVIEW DISCUSSION GUIDE: SCHOOL MANAGEMENT COMMITTEES (SMCs) BENEFICIARY SCHOOLS Respondent Data GI(P)1. Facilitator name: _________________ GI(P)2. Facilitator ID: ___________________ GI(P)3. Date: _________________________ GI(P)4. Respondent name: _______________ GI(P)5. School:________________________ GI(P)6. Village: ________________________ GI(P)7. Union Council: __________________ GI(P)8. District: _______________________ GI(P)9. Intervention type: (circle all that apply, to be completed prior to the interview) a. Community engagement/mobilization b. School construction/rehabilitation c. School consolidation d. Public-private partnership/ EMO e. Small grant GI(P)10. Respondent profile: [Enter designation and length of participation on school management committee (SMC). If possible, this section to be completed prior to the interview.] Respondent 1: Designation: Length of participation: Gender: Respondent 5: Designation: Length of participation: Gender: Respondent 2: Designation: Length of participation: Gender: Respondent 6: Designation: Length of participation: Gender: Respondent 3: Designation: Length of participation: Gender: Respondent 7: Designation: Length of participation: Gender: Respondent 4: Designation: Length of participation: Gender: Respondent 8: Designation: Length of participation: Gender: Sindh Community Mobilization Program – Midterm Performance Evaluation 75 GI(P)11. Starting time: ________________________ GI(P)12. End time: ___________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the effectiveness of community mobilization activities in your communities. Specifically, we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(S)13. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(S)14. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(S)15. When was SMC formed in this school? (Ask secretary)? SECTION B: EFFECTIVENESS OF CMP INTERVENTION FOR INCREASING ENROLLMENT/RETENTION/ATTENDANCE GI(S)16. Has this SMC conducted activities to increase enrollment, retention, or attendance? If yes, did CMP support these activities? [Probe how.] GI(S)17. In your opinion or experience, have these activities been successful in increasing enrollment, retention, or attendance? Explain how or why not. SECTION C: EFFECTIVENESS OF COMMUNITY ENGAGEMENT ACTIVITIES GI(S)18. Has CMP provided technical assistance to this SMC to carry out its functions? If yes, please explain. Has this technical assistance affected the SMC’s processes and/or capacity to carry out these functions? Explain how or why not. [Probe formal and informal technical assistance/support.][Probe types of formal technical assistance, including SIP, recordkeeping, budget management, small grant proposal design and management, training on roles and responsibilities, etc.] [Distinguish between what SMC is doing and what CMP is doing.] Sindh Community Mobilization Program – Midterm Performance Evaluation 76 GI(S)19. Has CMP provided technical assistance to one or more sub-committees to carry out their functions? If yes, please explain. Has this technical assistance affected the sub-committees’ processes and/or capacity to carry out these functions? Explain how or why not. [Probe formal and informal technical assistance/support.][Probe types of formal technical assistance, including development of TORs, training, etc.] [Distinguish between what SMC is doing and what CMP is doing.] GI(S)20. Has CMP’s support to this SMC affected how the SMC interacts with the community? Explain how or why not? [Probe community participation in meetings, community participation in SIP, sharing information, budget approval, how the SMC responds to community concerns about school management, e.g. teacher absenteeism, etc.] GI(S)21. Has this SMC or an SMC subcommittee developed and/or implemented a SIP/SMP? If yes, has CMP’s support to the SMC affected how the SMC engages/consults with community members in the SIP/SMP development or implementation process? Explain how or why not. GI(S)22. Have you discussed school matters with parents or other members of the community outside formal SMC meetings? Please explain. [Probe for specific examples: when, where, for what purpose] Respondent Response (yes/no) 1 2 3 4 5 6 7 8 GI(S)23. Has your interaction with the community been effective or ineffective for resolving issues related to enrollment or retention? (Probe for direct and indirect effects) Has parents’ responsiveness to these issues increased, decreased, or stayed the same in the past two years? How and why? Give examples. GI(S)24. What challenges, if any, has this SMC faced when interacting with the community? Explain with examples. [Probe general body meetings, SIP/SMP development process, enrolment campaigns, other outreach activities] [Probe how SMC responds to the identified challenges] GI(S)25. Has CMP’s support to this SMC affected how the SMC interacts with the district/tehsil education office? Explain how or why not? [Probe for frequency, who initiates the interaction, what, how and why/why not] Sindh Community Mobilization Program – Midterm Performance Evaluation 77 GI(S)26. What challenges, if any, has this SMC faced when interacting with the district/tehsil education office? Explain with examples. GI(S)27. Do you plan to continue community engagement activities supported by CMP after CMP ends? If yes, what activities would you like to continue? How? [Probe how, including training of new SMC members, conducting meetings with parents/community, resource mobilization for activities etc.] SECTION D: CONSOLIDATION (Ask only for consolidation schools) GI(S)28. Has community support for school consolidation changed over time? If yes, how and why? [Probe whether support increased, decreased, or stayed the same; probe effect of CMP.] GI(S)29. How, if at all, has this SMC been involved in the school consolidation process (before, during, and after consolidation)? Explain. [Probe two-way interaction between SMC, school, and district/tehsil education office.] GI(S)30. Did this SMC face any challenges during consolidation and how, if at all, did the SMC address those challenges? Give examples. GI(S)31. (ASK IF ANSWER TO Q30 IS YES) Has CMP provided support to this SMC to address challenges relating to the school consolidation process (before, during, and after consolidation)? Explain how or why. SECTION E: PUBLIC-PRIVATE PARTNERSHIP (Ask only for EMO schools) GI(S)32. Has community support for EMO school management changed over time? If yes, how and why? [Probe whether increased, decreased, or stayed the same.] GI(S)33. How, if at all, has this SMC been involved in the transition to EMO management (before, during, and after this transition)? Explain. [Probe two-way interaction between SMC, school, and district/tehsil education office.] GI(S)34. Did this SMC face any challenges during the transition to EMO management and how, if at all, did the SMC address those challenges? Give examples. GI(S)35. (ASK IF ANSWER TO Q29 IS YES) Has CMP provided support to this SMC to address challenges relating to the transition to EMO management (before, during, and after consolidation)? Explain how or why. Sindh Community Mobilization Program – Midterm Performance Evaluation 78 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (DISTRICT EDUCATION OFFICERS) Respondent Data GI(P)1. Facilitator name: ______________________________________________ GI(P)2. Facilitator ID: ________________________________________________ GI(P)3. Date: ______________________________________________________ GI(P)4. Respondent name and designation:_________________________________ GI(P)5. District: ____________________________________________________ GI(P)6. Tehsil: _____________________________________________________ GI(P)7. Starting time: ________________________________________________ GI(P)8. End time: ___________________________________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)9. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)10. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: RESPONDENT BACKGROUND (ask all respondents) GI(P)11. Years serving in your current position (in years)? [IF LESS THAN ONE YEAR, RECORD NUMBER OF MONTHS] years, months GI(P)12. Years serving in your current district (in years)? [IF LESS THAN ONE YEAR, RECORD NUMBER OF MONTHS] years, months Sindh Community Mobilization Program – Midterm Performance Evaluation 79 SECTION C: FACTORS AFFECTING ENROLLMENT GI(P)13. In your experience, what are the primary factors motivating parents’ decision to enroll or keep their children in school? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] GI(P)14. In your experience, what are the primary barriers to parents’ decision to enroll or keep their children in school? [Probe for drop-out and never enrolled, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] SECTION D: EFFECT OF CMP ON ENROLLMENT GI(P)15. Does your office conduct or support any community mobilization activities designed to increase enrollment? Explain? GI(P)16. Has CMP contributed to these mobilization activities? If yes, how? GI(P)17. Are you familiar with other CMP activities to encourage enrollment and retention in your district? [Probe for specific CMP activities.] [Probe separately for enrollment and retention.] GI(P)18. (IF YES TO 17) In your opinion, have these activities affected parents’ decision to enroll or keep their children in school? If yes, how? If no, why not? [Probe separately for enrollment and retention, gender, what, how and why] [Insert relevant intervention: SMC, new school, consolidation, other.] GI(P)19. (IF YES TO 17) In your experience, what kinds of activities are most effective in increasing enrollment or retention and why? [Probe gender, what, how and why.] SECTION E: DEO ENGAGEMENT WITH SCHOOLS AND SMCS GI(P)20. Do you interact with SMCs and/or SMC sub-committees? If yes, for what purpose? [Probe for how often, who initiates, how and why.] [Probe for interaction specific to increasing enrollment.] [Probe for role of district education department in SIP/SMP.] GI(P)21. (ASK IF ANSWER TO Q20 IS POSITIVE) Has your interaction with SMCs and/or SMC sub-committees changed over the past two years? If yes, how? [Probe for frequency, purpose, who initiates, how and why.] [Probe for interaction specific to increasing enrollment] [Probe for role of district education department in SIP/SMP.] Sindh Community Mobilization Program – Midterm Performance Evaluation 80 GI(P)22. Has your interaction with schools changed over the past two years? If yes, did CMP contribute to the change? If yes, how? If no, why not? [Probe for frequency, purpose, who initiates, how and why] [Probe for interaction specific to increasing enrollment.] SECTION F: CONSOLIDATION (Ask only for consolidation schools) Needs Identification and Assessment GI(P)23. Did CMP provide any technical assistance to your office to implement the consolidation policy? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important)] [Probe whether technical assistance is one-off or regular.] GI(P)24. (ASK IF Q23 IS POSITIVE, i.e. if DEO received TA from CMP) Did CMP address your technical assistance needs for school consolidation? How, explain. [Probe what, how and why/why not.] Effectiveness of CMP Technical Assistance GI(P)25. How, if at all, has the technical assistance provided by CMP affected your office’s implementation of the consolidation policy? Explain. [Probe what, how; e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.] GI(P)26. (ASK IF ANSWER TO Q24 IS NEGATIVE) What, if any, barriers did you experience to using CMP technical assistance to support the consolidation process? [Probe for financial, political, administrative, social barriers, other possible barriers; e.g. accessing TA, applying products or recommendations from TA, etc.] Institutionalization and Sustainability GI(P)27. Has your office made any practices promoted by CMP to support school consolidation part of your normal practice? If yes, how and to what extent? Please provide examples. [Probe specific practices, e.g. school selection, community consultation, etc., and how (process of institutionalization.] GI(P)28. How likely is your office to continue applying practices promoted by CMP to support consolidation after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] Sindh Community Mobilization Program – Midterm Performance Evaluation 81 SECTION G: PUBLIC-PRIVATE PARTNERSHIP (Ask only for EMO schools) Needs Identification and Assessment GI(P)29. Did CMP provide any technical assistance to your office to implement EMO policies? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important)] [Probe whether technical assistance is one-off or regular.] GI(P)30. (ASK IF Q29 IS POSITIVE, i.e. if DEO received TA from CMP) Did CMP address your technical assistance needs relating to EMOs? How, explain. [Probe what, how and why/why not.] Effectiveness of CMP Technical Assistance GI(P)31. How, if at all, has the technical assistance provided by CMP affected your office’s implementation of EMO policies? Explain. [Probe what, how; e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.] GI(P)32. (ASK IF ANSWER TO Q31 IS NEGATIVE) What, if any, barriers did you experience to using CMP technical assistance to support EMO policies? [Probe for financial, political, administrative, social barriers, other possible barriers; e.g. accessing TA, applying products or recommendations from TA, etc.] Institutionalization and Sustainability GI(P)33. Has your office made any practices promoted by CMP to support EMOs part of your normal practice? If yes, how and to what extent? Please provide examples. [Probe specific practices, e.g. school selection, community consultation, etc., and how (process of institutionalization).] GI(P)34. How likely is your office to continue applying practices promoted by CMP to support EMOs after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] Sindh Community Mobilization Program – Midterm Performance Evaluation 82 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (EMOs) ***Questions should pertain specifically to schools where EMO is active*** Respondent Data GI(P)1. Facilitator name: _________________ GI(P)2. Facilitator ID: ___________________ GI(P)3. Date: _________________________ GI(P)4. Respondent name: _______________ GI(P)5. School:________________________ GI(P)6. Village: ________________________ GI(P)7. Union Council: __________________ GI(P)8. District: _______________________ GI(P)9. Intervention type: (can be multiple) a. Community engagement/ mobilization b. School construction/rehabilitation c. School consolidation d. Public-private partnership/ EMO e. Small grant/neighboring GI(P)10. Starting time: ________________________ GI(P)11. End time: ___________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)12. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)13. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: BACKGROUND GI(P)14. Are you aware of any activities supported by the Community Mobilization Program (CMP) to increase community engagement and/or increase enrollment or retention in your school? If yes, describe the activities. [If necessary, mention IPs by name] [Probe for activity type and process of activities.] Sindh Community Mobilization Program – Midterm Performance Evaluation 83 SECTION C: EFFECT OF CMP ON SCHOOL MANAGEMENT IN EMO SCHOOLS GI(P)15. In your experience, has school management changed since the transition to EMO management? If yes, how? [Probe mechanisms/outcomes relating to oversight, accountability, transparency, etc.] [Probe for enrollment and retention: what, how and why.] GI(P)16. In your experience, what have been the biggest challenges transitioning to EMO school management? How have you addressed these challenges? GI(P)17. In your opinion, have CMP activities affected school management in your school(s)? If so, what effect did CMP activities have on school management? [Probe separately for enrollment and retention, gender, what, how and why] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] GI(P)18. In your opinion, how can CMP improve its support for community engagement and/or school management? [Ask for specific activities for boys and girls: what, how and why.] SECTION D: FACTORS AFFECTING ENROLLMENT GI(P)19. In your experience, what are the primary factors that affect parents’ decision to enroll their child/children in school in this community? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] GI(P)20. In your experience, what are the primary factors that affect parents’ decision to keep their child/children in school in this community? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] SECTION E: EFFECT OF CMP ON ENROLLMENT IN EMO SCHOOLS GI(P)21. In your opinion or experience, have CMP activities in your community affected parents’ decisions about enrolling their child/children in school or keeping them in school? Explain how and why, provide examples? [Probe for specific CMP activities and how or why not for differential effects for different types of children, including gender and age.] Sindh Community Mobilization Program – Midterm Performance Evaluation 84 SECTION F: EFFECT OF CMP ON COMMUNITY ENGAGEMENT IN EMO SCHOOLS GI(P)21. Does your school pro-actively engage with communities to increase enrollment or retention? How, please explain. [Probe for type of activity, frequency and process of activities] GI(P)22. (ASK IF ANSWER TO Q21 IS YES) In your opinion, have CMP activities had any effect on community engagement activities in your community? What effect did they have and how? [Probe gender, how and why.] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] GI(P)23. How, if at all, has community support for or awareness of EMOs changed since the transition to EMOs? How? [Probe for head teacher and parents separately: what, how and why/why not.] GI(P)24. How, if at all, has support for or awareness of EMOs affected enrollment and retention? If it has affected enrollment or retention, how has it done so? Give examples. [Probe what, how and why.] GI(P)25. In your opinion, have CMP activities had any effect on community support for or awareness of EMOs in your community? What effect did they have and how? [Probe gender, how and why.] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] SECTION G: TECHNICAL ASSISTANCE FOR EMO MANAGEMENT GI(P)26. Have you received technical assistance from CMP to support the transition to EMO management? [Probe what, how and frequency.] GI(P)27. (ASK IF ANSWER TO Q26 IS YES) If yes, how has the technical assistance supported the transition to EMO management? Give examples. [Probe what, how and why/why not.] Sindh Community Mobilization Program – Midterm Performance Evaluation 85 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (EMOs) ***Questions should pertain specifically to schools where EMO is active*** Respondent Data GI(P)1. Facilitator name: _________________ GI(P)2. Facilitator ID: ___________________ GI(P)3. Date: _________________________ GI(P)4. Respondent name: _______________ GI(P)5. School:________________________ GI(P)6. Village: ________________________ GI(P)7. Union Council: __________________ GI(P)8. District: _______________________ GI(P)9. Intervention type: (can be multiple) a. Community engagement/ mobilization b. School construction/rehabilitation c. School consolidation d. Public-private partnership/ EMO e. Small grant/neighboring GI(P)10. Starting time: ________________________ GI(P)11. End time: ___________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)12. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)13. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: BACKGROUND GI(P)14. Are you aware of any activities supported by the Community Mobilization Program (CMP) to increase community engagement and/or increase enrollment or retention in your school? If yes, describe the activities. [If necessary, mention IPs by name] [Probe for activity type and process of activities.] Sindh Community Mobilization Program – Midterm Performance Evaluation 86 SECTION C: EFFECT OF CMP ON SCHOOL MANAGEMENT IN EMO SCHOOLS GI(P)15. In your experience, has school management changed since the transition to EMO management? If yes, how? [Probe mechanisms/outcomes relating to oversight, accountability, transparency, etc.] [Probe for enrollment and retention: what, how and why.] GI(P)16. In your experience, what have been the biggest challenges transitioning to EMO school management? How have you addressed these challenges? GI(P)17. In your opinion, have CMP activities affected school management in your school(s)? If so, what effect did CMP activities have on school management? [Probe separately for enrollment and retention, gender, what, how and why] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] GI(P)18. In your opinion, how can CMP improve its support for community engagement and/or school management? [Ask for specific activities for boys and girls: what, how and why.] SECTION D: FACTORS AFFECTING ENROLLMENT GI(P)19. In your experience, what are the primary factors that affect parents’ decision to enroll their child/children in school in this community? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] GI(P)20. In your experience, what are the primary factors that affect parents’ decision to keep their child/children in school in this community? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] SECTION E: EFFECT OF CMP ON ENROLLMENT IN EMO SCHOOLS GI(P)22. In your opinion or experience, have CMP activities in your community affected parents’ decisions about enrolling their child/children in school or keeping them in school? Explain how and why, provide examples? [Probe for specific CMP activities and how or why not for differential effects for different types of children, including gender and age.] Sindh Community Mobilization Program – Midterm Performance Evaluation 87 SECTION F: EFFECT OF CMP ON COMMUNITY ENGAGEMENT IN EMO SCHOOLS GI(P)21. Does your school pro-actively engage with communities to increase enrollment or retention? How, please explain. [Probe for type of activity, frequency and process of activities] GI(P)22. (ASK IF ANSWER TO Q21 IS YES) In your opinion, have CMP activities had any effect on community engagement activities in your community? What effect did they have and how? [Probe gender, how and why.] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] GI(P)23. How, if at all, has community support for or awareness of EMOs changed since the transition to EMOs? How? [Probe for head teacher and parents separately: what, how and why/why not.] GI(P)24. How, if at all, has support for or awareness of EMOs affected enrollment and retention? If it has affected enrollment or retention, how has it done so? Give examples. [Probe what, how and why.] GI(P)25. In your opinion, have CMP activities had any effect on community support for or awareness of EMOs in your community? What effect did they have and how? [Probe gender, how and why.] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] SECTION G: TECHNICAL ASSISTANCE FOR EMO MANAGEMENT GI(P)26. Have you received technical assistance from CMP to support the transition to EMO management? [Probe what, how and frequency.] GI(P)27. (ASK IF ANSWER TO Q26 IS YES) If yes, how has the technical assistance supported the transition to EMO management? Give examples. [Probe what, how and why/why not.] Sindh Community Mobilization Program – Midterm Performance Evaluation 88 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (HEAD TEACHERS) BENEFICIARY SCHOOLS Respondent Data GI(P)1. Facilitator name: _________________ GI(P)2. Facilitator ID: ___________________ GI(P)3. Date: _________________________ GI(P)4. Respondent name: _______________ GI(P)5. School:________________________ GI(P)6. Village: ________________________ GI(P)7. Union Council: __________________ GI(P)8. District: _______________________ GI(P)9. Intervention type: (can be multiple) a. Community engagement/ mobilization b. School construction/rehabilitation c. School consolidation d. Public-private partnership/ EMO e. Small grant/neighboring GI(P)10. Starting time: ________________________ GI(P)11. End time: ___________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)12. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)13. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: RESPONDENT BACKGROUND (ASK ALL RESPONDENTS) GI(P)14. Gender (CIRCLE ONE NUMBER) 1 Male 2 Female Sindh Community Mobilization Program – Midterm Performance Evaluation 89 GI(P)15. For how many years have you worked in government schools? __________________ GI(P)16. For how many years have you worked in this school? ___________________ GI(P)17. For how many years have you been working as a head teacher/principal? _________________ GI(P)18. Were you the head teacher/principal before [consolidation/transition to EMO management] in this school? [If applicable] 1 Yes 2 No GI(P)19. What is the highest level of education you have completed? [CIRCLE ONE NUMBER] 1 Matric 2 F.A/FSc. 3 Technical education (diploma) 4 BA/BSc 5 MA/ MSc. or above 6 Other (e.g. religious education) (Specify ______________________ ) 7 Refused to answer SECTION C: FACTORS AFFECTING ENROLLMENT GI(P)20. In your experience, what are the primary factors that encourage parents to enroll or keep their children in school in this community? [Probe for new and sustained enrollment, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] GI(P)21. In your experience, what are the primary barriers that parents face enrolling or keeping their children in school in this community? [Probe for drop-out and never enrolled, what, how and why.] [Differentiate with gender lens between external factors (poverty, distance, gender role) and internal factors (school improvement, participation, awareness).] GI(P)22. Has enrollment increased in the last two years? If yes, what challenges, if any, does this school face due to increased enrollment? How have you addressed these challenges? SECTION D: EFFECT OF CMP ON ENROLLMENT GI(P)23. Does your school pro-actively engage with communities to increase enrollment or retention? Please explain. [Probe for type of activity, frequency and process of activities.] Sindh Community Mobilization Program – Midterm Performance Evaluation 90 GI(P)24. In your opinion, do these activities influence parents’ decisions to enroll or keep their children in school? If yes, how, please explain. [Probe separately for enrollment and retention, gender, what, how and why.] GI(P)25. Are you aware of any activities supported by the Community Mobilization Program (CMP) that help increase enrollment or retention or improve school management in your school? If yes, describe the activities. [If necessary, mention IPs by name] [Probe for activity type and process of activities.] GI(P)26. (ASK IF ANSWER TO Q25 IS YES) In your opinion, have CMP activities had any effect on enrollment or retention? What effect did they have and how? [Probe separately for enrollment and retention, gender, what, how and why.] [Probe relevant activities: engagement, SMC, new school, consolidation, other.] GI(P)27. In your experience, have CMP activities had any effect on school management? If yes, please explain. [Probe mechanisms/outcomes relating to oversight, accountability, transparency, etc.] [Probe for enrollment and retention: what, how and why.] GI(P)28. In your opinion, how can CMP improve its support for activities to increase enrollment and retention or improve school management? [Ask for specific activities for boys and girls: what, how and why.] SECTION E: SCHOOL ENGAGEMENT WITH COMMUNITY GI(P)29. How do parents engage with your school? [Probe reasons, process, frequency] GI(P)30. How does your school engage with parents? [Probe specifically for parent-teacher meetings, reasons, process and frequency.] GI(P)31. Has interaction between parents and your school changed since CMP’s intervention? If yes, how? [Probe specifically for parent-teacher meeting: reasons, process and frequency.] GI(P)32. What, if any, challenges do you face in interacting with parents? If so, how do you address challenges in interaction? GI(P)33. Has your interaction with the SMC changed since CMP intervention? If yes, how? [Probe specifically for parent-teacher meeting: reasons, process and frequency.] Sindh Community Mobilization Program – Midterm Performance Evaluation 91 GI(P)34. What, if any, challenges do you face in interacting with the SMC? If so, how do you address challenges in interaction? GI(P)35. Does interaction between parents and your school have any effect on school management? [Probe what, how and why] [Probe for child performance, quality of instruction, other management mechanisms/outcomes such as accountability, transparency, staffing, etc.] SECTION F: CONSOLIDATION (Ask only in consolidation schools) GI(P)36. Details of school consolidation: [E.g. cluster, physical consolidation, etc. – Were schools located in same premises prior to consolidation?] GI(P)37. As the principal/head teacher, have you received technical assistance from CMP to support school consolidation? [Probe what, how and frequency.] GI(P)38. (ASK IF ANSWER TO Q37 IS YES) If yes, how has the technical assistance supported the consolidation process? Give examples. [Probe what, how and why/why not.] GI(P)39. Has community support for or awareness of consolidation changed in the last two years? How? [Probe for head teacher and parents separately: what, how and why/why not.] GI(P)40. (ASK IF Q39 IS YES) How, if at all, has support for or awareness of the consolidation process affected enrollment and retention? If it has affected enrollment or retention, how has it done so? Give examples. [Probe what, how and why.] SECTION G: PUBLIC-PRIVATE PARTNERSHIP (ASK ONLY IN EMO SCHOOLS) GI(P)41. As the principal/head teacher, have you received technical assistance from CMP to support the transition to EMO management? [Probe what, how and frequency.] GI(P)42. (ASK IF ANSWER TO Q41 IS YES) If yes, how has the technical assistance supported the transition to EMO management? Give examples. [Probe what, how and why/why not.] Sindh Community Mobilization Program – Midterm Performance Evaluation 92 GI(P)43. Has community support for or awareness of EMOs changed in the last two years? How? [Probe for head teacher and parents separately: what, how and why/why not.] GI(P)44. (ASK IF Q43 IS YES) How, if at all, has support for or awareness of EMOs affected enrollment and retention? If it has affected enrollment or retention, how has it done so? Give examples. [Probe what, how and why.] Sindh Community Mobilization Program – Midterm Performance Evaluation 93 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (IRD District Staff) Respondent Data KI(IP)1. Facilitator name:____________________________________________ KI(IP)2. Facilitator ID:______________________________________________ KI(IP)3. Date:____________________________________________________ KI(IP)4. Respondent name and designation: ______________________________ KI(IP)5. District:__________________________________________________ KI(IP)6. Starting time: ______________________________________________ KI(IP)7. End time: _________________________________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. KI(IP)8. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No KI(IP)9. Do you give permission to record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: RESPONDENT BACKGROUND (ask all respondents) KI(IP)10. For how many years have you been in your current position (in years)? ________________ KI(IP)11. How would you describe your roles and responsibilities in CMP? SECTION C: EFFECT OF CMP ON ENROLLMENT KI(IP)12. In your opinion, which, if any, CMP interventions/services were most effective in increasing enrollment or retention? Why were they effective? [Probe gender, what, how and why.] KI(IP)13. What are the main challenges you have faced implementing CMP activities? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain. Sindh Community Mobilization Program – Midterm Performance Evaluation 94 SECTION D: ENROLLMENT AS INDICATOR FOR ACCESS KI(IP)14. In your opinion, is change in enrollment a meaningful indicator by which to measure the performance of CMP? Why or why not? [Probe why/why not] [Probe whether change in enrollment is sufficient to measure CMP performance.] [Probe attendance, retention.] KI(IP)15. In your opinion, is change in enrollment a meaningful indicator by which to measure access to education? Why or why not? [Probe why/why not.] [Probe whether change in enrollment is sufficient to measure access to education.] [Probe attendance, retention.] KI(IP)16. In your opinion, do CMP’s methods for measuring change in enrollment accurately measure change in enrollment? Why or why not? [Probe data reliability, credibility, precision, timeliness, etc.] KI(IP)17. At any time have you exceeded your enrollment target? Why? At any time have you failed to meet your enrollment target? Why? [Probe whether estimate was appropriate, implementation factors, external factors, etc.] KI(IP)18. Based on your experience, are CMP enrollment targets appropriate for achieving program objectives? Why or why not? [Probe whether targets set reasonable expectations, whether targets can reasonably be met, etc.] KI(IP)19. Have you been consulted when setting or adjusting enrollment targets? If so, what feedback did you provide? Were your inputs or feedback addressed? If so, how? If not, why not? [Probe whether consultation is regular; probe result of target adjustment.] KI(IP)20. Based on your experience, how can targets or the process for setting targets be improved? KI(IP)21. Are there other indicators, in addition to enrollment, that would be meaningful indicators of the performance of CMP? Please explain. [Probe what, how and why; probe other ways to measure enrollment; probe attendance, retention as possible access indicators.] KI(IP)22. (If Q17 is yes) Can these indicators feasibly be measured? If so, how? [Probe how; probe cost, data sensitivity, availability of data, where else indicators are being tracked, etc.] Sindh Community Mobilization Program – Midterm Performance Evaluation 95 KI(IP)23. Are there other indicators, in addition to enrollment, that would be meaningful indicators of improved access to schooling? Please explain. [Probe what, how and why; probe other ways to measure enrollment; probe attendance, retention as possible access indicators.] [If different than above, probe if indicator is appropriate for measuring CMP performance.] KI(IP)24. If Q19 is yes: Can these indicators feasibly be measured? If so, how? [Probe how; probe cost, data sensitivity, availability of data, where else indicators are being tracked, etc.] SECTION E: CMP ENGAGEMENT WITH SMC, SCHOOL, AND COMMUNITY KI(IP)25. What training or support have you provided to SMCs to help them better carry out their activities? [Probe what, how and why.] [List all activities.] KI(IP)26. How, if at all, do you identify or assess training or capacity building needs for SMCs? Provide examples. KI(IP)27. How, if at all, have the activities of SMCs changed since the launch of CMP? Give examples. [Probe community engagement, SIP/SMP, financial management, spending decisions, school monitoring, etc.; probe how and why.] KI(IP)28. What challenges have you faced engaging with SMCs? How has CMP addressed these challenges? Was your strategy for addressing these challenges effective? Explain. [Probe for SMC and community: gender, what, how and why.] KI(IP)29. What challenges have you faced engaging with schools and/or head teachers? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain. [Probe for SMC and community: gender, what, how and why.] KI(IP)30. Do you engage directly with parents/communities? If so, how and to what extent? KI(IP)31. IF Q27 is yes: What challenges have you faced engaging directly with communities? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain. [Probe for SMC and community: gender, what, how and why.] Sindh Community Mobilization Program – Midterm Performance Evaluation 96 SECTION F: CMP ENGAGEMENT WITH DEOS/TO KI(IP)32. What training or support have you provided to education managers (district or taluka education offices) to help them better carry out their activities? [Probe what, how and why.] [List all activities.] KI(IP)33. How do you identify or assess training or capacity building needs for district or taluka education offices? KI(IP)34. How, if at all, have the activities/operating procedures of education managers changed since the launch of CMP? Give examples. [Probe community engagement, SIP/SMP, financial management, spending decisions, school monitoring, etc.; probe how and why] KI(IP)35. What challenges have you faced in engaging with district or taluka education offices? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain. [Probe for SMC and community: gender, what, how and why.] SECTION G: CONSOLIDATION KI(IP)36. What challenges have you faced during the consolidation process? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain. [Probe what, how and why.] KI(IP)37. How, if it all, do CMP activities change following school consolidation? [Probe what, how and why; probe interaction with community and SMCs.] SECTION H: PUBLIC-PRIVATE PARTNERSHIP KI(IP)38. What challenges have you faced during the transition to EMO school management? How have you addressed these challenges? Was your strategy for addressing these challenges effective? Explain and give examples. [Probe what, how and why.] KI(IP)39. How, if at all, do CMP’s activities changed following the transition to EMO school management? [Probe what, how and why; probe interaction with community and SMCs.] Sindh Community Mobilization Program – Midterm Performance Evaluation 97 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (SCHOOL EDUCATION AND LITERACY DEPARTMENT, PPP NODE) Respondent Data GI(P)1. Facilitator name: ______________________________________________ GI(P)2. Facilitator ID: ________________________________________________ GI(P)3. Date: ______________________________________________________ GI(P)4. Respondent name and designation:_________________________________ GI(P)5. Starting time: ________________________________________________ GI(P)6. End time: ___________________________________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)7. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)8. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION A: RESPONDENT BACKGROUND (ask all respondents) GI(P)9. How long have you been serving in your current position (in years)? ____________________ [If respondent doesn’t know how many years ago, ask which year respondent began.] GI(P)10. What activities, if any, does the PPP node undertake to increase enrollment/retention? What activities, if any, does the PPP node undertake with respect to community engagement? GI(P)11. How would you describe the PPP node’s roles and responsibilities for coordinating CMP activities with respect to EMOs? Sindh Community Mobilization Program – Midterm Performance Evaluation 98 SECTION B: PUBLIC-PRIVATE PARTNERSHIPS Needs Identification and Assessment GI(P)12. Did CMP provide any technical assistance to your office to implement and/or coordinate EMO activities? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important).] [Probe whether technical assistance is one-off or regular.] GI(P)13. (ASK IF Q12 IS POSITIVE, i.e. if respondent received TA from CMP) Did CMP address your technical assistance needs related to EMOs? How? Explain. [Probe what, how and why/why not.] Effectiveness of CMP Technical Assistance GI(P)14. How, if at all, has the technical assistance provided by CMP affected the department’s implementation and/or coordination of EMO activities? Explain. [Probe what, how (e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.)] GI(P)15. (ASK IF ANSWER TO Q14 IS NEGATIVE) What, if any, are the barriers did you experience using CMP technical assistance to support implementation and/or coordination of EMO activities? [Probe for financial, political, administrative, social barriers, other possible barriers (e.g. accessing TA, applying products or recommendations from TA, etc.).] Institutionalization and Sustainability GI(P)16. Has your office made any practices promoted by CMP to support EMO activities part of your policies or standard practices? If yes, how and to what extent? Please provide examples. [Probe specific practices (e.g. school selection, community consultation, etc.) and how (e.g. process of institutionalization).] GI(P)17. How likely is your office to continue applying practices promoted by CMP to support EMO activities after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] Sindh Community Mobilization Program – Midterm Performance Evaluation 99 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (SCHOOL EDUCATION AND LITERACY DEPARTMENT - RSU) Respondent Data GI(P)1. Facilitator name: ______________________________________________ GI(P)2. Facilitator ID: ________________________________________________ GI(P)3. Date: ______________________________________________________ GI(P)4. Respondent name and designation:_________________________________ GI(P)5. Starting time: ________________________________________________ GI(P)6. End time: ___________________________________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)7. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)8. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: RESPONDENT BACKGROUND GI(P)9. How long have you been serving in your current position (in years)? ____________________ (If respondent doesn’t know how many years ago, ask which year respondent began) GI(P)10. What activities does RSU undertake to increase enrollment/retention? What activities does RSU undertake with respect to community engagement? GI(P)11. How would you describe RSU’s roles and responsibilities for coordinating CMP activities? Sindh Community Mobilization Program – Midterm Performance Evaluation 100 SECTION C: CONSOLIDATION Needs Identification and Assessment GI(P)12. Did CMP provide any technical assistance to your office to implement the consolidation policy and/or coordinate consolidation activities? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important).] [Probe whether technical assistance is one-off or regular.] GI(P)13. (ASK IF Q12 IS POSITIVE, i.e. if SELD received TA from CMP) Did CMP address your technical assistance needs for school consolidation? How? Explain. [Probe what, how and why/why not] Effectiveness of CMP Technical Assistance GI(P)14. How, if at all, has the technical assistance provided by CMP affected RSU’s implementation of the Consolidation Policy and/or coordination of consolidation activities? Explain. [Probe what, how (e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.).] GI(P)15. (ASK IF ANSWER TO Q14 IS NEGATIVE) What, if any, barriers did you experience to using CMP technical assistance to support the consolidation process? [Probe for financial, political, administrative, social barriers, other possible barriers (e.g. accessing TA, applying products or recommendations from TA, etc.).] Institutionalization and Sustainability GI(P)16. Has your office made any practices promoted by CMP to support school consolidation part of your standard practices? If yes, how and to what extent? Please provide examples. [Probe specific practices (e.g. school selection, community consultation, etc.) and how (process of institutionalization, etc.).] GI(P)17. How likely is your office to continue applying practices promoted by CMP to support consolidation after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] Sindh Community Mobilization Program – Midterm Performance Evaluation 101 KEY INFORMANT INTERVIEW DISCUSSION GUIDE (SCHOOL EDUCATION AND LITERACY DEPARTMENT) Respondent Data GI(P)1. Facilitator name: ______________________________________________ GI(P)2. Facilitator ID: ________________________________________________ GI(P)3. Date: ______________________________________________________ GI(P)4. Respondent name and designation:_________________________________ GI(P)5. Starting time: ________________________________________________ GI(P)6. End time: ___________________________________________________ SECTION A: INTRODUCTION Assalam-o-Alaikum (Greetings); My name is . I work for Management Systems International, an international consultancy contracted by USAID, and I am here to collect data to assess the benefits of community mobilization activities designed to increase enrollment and improve school management in your communities. Specifically we are interested in the effectiveness of community engagement activities, school construction and/or rehabilitation, school consolidation, and private-sector management of government schools. We would like to ask you a series of questions about the impact of these projects. This interview will take approximately one hour to 90 minutes to complete. GI(P)7. Do you give permission to continue with this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No GI(P)8. Do you give permission to audio record this interview? (CIRCLE ONE NUMBER) 1 Yes 2 No SECTION B: RESPONDENT BACKGROUND (ask all respondents) GI(P)9. How long have you been serving in your current position (in years)? ____________________ [If respondent doesn’t know how many years ago, ask which year respondent began.] GI(P)10. How would you describe your department’s policy and strategy for increasing enrollment/retention? GI(P)11. How would you describe your department’s policy and strategy for engaging communities? Sindh Community Mobilization Program – Midterm Performance Evaluation 102 SECTION C: EFFECT OF CMP ON ENROLLMENT GI(P)12. Are you familiar with CMP activities to encourage enrollment and retention in Sindh? [Probe for specific CMP activities.] [Probe separately for enrollment and retention.] GI(P)13. (IF YES TO 12) In your opinion, have these activities affected parents’ decision to enroll or keep their children in school? If yes, how? If no, why not? [Probe separately for enrollment and retention, gender, what, how and why.] [Insert relevant intervention: SMC, new school, consolidation, other.] GI(P)14. (IF YES TO 12) In your experience, what kinds of activities are most effective in increasing enrollment or retention and why? [Probe gender, what, how and why.] SECTION D: CONSOLIDATION Needs Identification and Assessment GI(P)15. Did CMP provide any technical assistance to your office to implement the consolidation policy? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important).] [Probe whether technical assistance is one-off or regular.] GI(P)16. (ASK IF Q15 IS POSITIVE, i.e. if SELD received TA from CMP) Did CMP address your technical assistance needs for school consolidation? How? Explain. [Probe what, how and why/why not.] Effectiveness of CMP Technical Assistance GI(P)17. How, if at all, has the technical assistance provided by CMP affected the department’s implementation of the Consolidation Policy? Explain. [Probe what, how (e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.).] GI(P)18. (ASK IF ANSWER TO Q24 IS NEGATIVE) What, if any, barriers did you experience to using CMP technical assistance to support the consolidation process? [Probe for financial, political, administrative, social barriers, other possible barriers (e.g. accessing TA, applying products or recommendations from TA, etc.).] Sindh Community Mobilization Program – Midterm Performance Evaluation 103 Institutionalization and Sustainability GI(P)19. Has your office made any practices promoted by CMP to support school consolidation part of your policy or standard practices? If yes, how and to what extent? Please provide examples. [Probe specific practices (e.g. school selection, community consultation, etc.) and how (process of institutionalization, etc.).] GI(P)20. How likely is your office to continue applying practices promoted by CMP to support consolidation after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] SECTION E: PUBLIC-PRIVATE PARTNERSHIP Needs Identification and Assessment GI(P)21. Did CMP provide any technical assistance to your office to implement EMO policies? If yes, did they consult you or assess your needs prior to providing assistance? [Probe what (specific needs, services, hardware, etc.), why (why are they important).] [Probe whether technical assistance is one-off or regular.] GI(P)22. (ASK IF Q21 IS POSITIVE, i.e. if SELD received TA from CMP) Did CMP address your technical assistance needs related to EMOs? How? Explain. [Probe what, how and why/why not.] Effectiveness of CMP Technical Assistance GI(P)23. How, if at all, has the technical assistance provided by CMP affected the department’s implementation of EMO policies? Explain. [Probe what, how (e.g. school selection, DEO compliance with consolidation policy, school compliance with consolidation policy, etc.).] GI(P)24. (ASK IF ANSWER TO Q23 IS NEGATIVE) What, if any, barriers did you experience to using CMP technical assistance to support the transition to EMO management? [Probe for financial, political, administrative, social barriers, other possible barriers (e.g. accessing TA, applying products or recommendations from TA, etc.).] Institutionalization and Sustainability GI(P)25. Has your office made any practices promoted by CMP to support EMO policies part of your policies or standard practices? If yes, how and to what extent? Please provide examples. [Probe specific practices (e.g. school selection, community consultation, etc.) and how (e.g. process of institutionalization, etc.).] Sindh Community Mobilization Program – Midterm Performance Evaluation 104 GI(P)26. How likely is your office to continue applying practices promoted by CMP to support consolidation after CMP ends? If likely, how and why? If not likely, why not? [Probe specific practices.] Sindh Community Mobilization Program – Midterm Performance Evaluation 105 Annex 6: Documents Reviewed CMP. (2014, January 31). Extended Quarterly Reports: Year One/Quarter One. CMP. (2014, April 30). Extended Quarterly Reports: Year One/Quarter Two. CMP. (2014, February). Revised Extended Annual Work Plan: Year One. CMP. (2014, May 31). Community Mobilization Baseline Report. CMP. (2014, May 31). Gender Analysis Report. CMP. (2014, May 31). Sindh Nutrition and Health Practices Survey Planning Report 2014. CMP. (2014, July 31). Extended Quarterly Reports: Year One/Quarter Three. CMP. (2014, September 30). Year II Annual Work Plan. CMP. (2015, January 31). Extended Quarterly Reports: Year Two/Quarter One. CMP. (2015, April 30). Extended Quarterly Reports: Year Two/Quarter Two. CMP. (2015, October 30). Extended Quarterly Reports: Year Two/Quarter Four. CMP. (2015, November 30). Year III Annual Work Plan. CMP. (2015, October 1 – 2016, September 30). Annual Report – Year III. CMP. (2016). Fact Sheet. CMP. (2016, January 31). Extended Quarterly Reports: Year Three/Quarter One. CMP. (2016, April 29). Extended Quarterly Reports: Year Three/Quarter Two. CMP. (2016, July 29). Extended Quarterly Reports: Year Three/Quarter Three. CMP. (2016, October 24). Year IV Work Plan. CMP. (2017, January 30). Extended Quarterly Reports: Year Four/Quarter One. Education & Literacy Department, Government of Sindh. (N.D.). Sindh Education Sector Plan 2014- 2018. Hirwani, Mumtaz. (2016, October 30). Data Portrayal of Increase/Decrease in Enrolment, particularly of Girls’ Students in CMP schools – Trend Analysis. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Dadu. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Jacobabad. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Kambar Shahdadkot. Sindh Community Mobilization Program – Midterm Performance Evaluation 106 IMMAP. (2013). Sindh Basic Education Programme District Education Profile: 5 Towns of Karachi. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Kashmore. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Khairpur. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Larkana. IMMAP. (2013). Sindh Basic Education Programme District Education Profile: Sukkur. Office of Inspector General. (2014, March 21). Audit of USAID/Pakistan’s Sindh Basic Education Program. Provincial Assembly of Sindh. (2010, February 18). The Sindh Public-Private Partnership Act, 2010. Public Private Partnership Unit – Finance Department, Government of Sindh. Concession Agreement between School Education Department – Government of Sindh and [Organization Name: ABC – XYZ]. Reform Support Unit, Education & Literacy Department, Government of Sindh. (2012, February 16). Notification: School Consolidation Policy/2011. School Education and Literacy Department, Government of Sindh. (2015, March 12). EMOs Notification: No. SO (G-1) E&L/SEMO/2015. School Education and Literacy Department and Public Private Partnership Unit – Finance Department, Government of Sindh. (2017, April 3). Invitation for Bids. Services, General, Administration and Co-ordination Department, Government of Sindh. (2015, February 25). District Support Team Terms of Reference: No. SO(C-IV) SGA&CD/4-22/14. Sindh Community Mobilization Program – Midterm Performance Evaluation 107 Annex 7: Sampling Tables 3 through 7 detail the population (i.e., total number of schools in Sindh, by district), sample frame (i.e., CMP schools and their nearest neighbor matches, by district), and the final sample (CMP and non￾CMP schools, by district), respectively, based on 2012-2013 SEMIS data. Of note, 2012-2013 SEMIS data was available for 399 CMP schools, for which 389 had matches that met the matching criteria. Forty￾eight schools matched to more than one CMP school. TABLE 3: POPULATION FRAME Districts Population Frame CMP Non-CMP Total EMOs Neighboring Other Sub-total Dadu 50 13 63 2,072 2,135 Jacobabad 37 13 50 1,419 1,469 Kashmore 42 16 58 1,436 1,494 Khairpur 3 41 12 56 3,410 3,466 Larkana 27 11 38 1,148 1,186 Qambar Shahdadkot 36 18 54 1,577 1,631 Sukkur 1 35 11 47 1,227 1,274 Districts (Total) 4 268 94 366 12,289 12,655 Bin Qasim Town 3 3 6 194 200 Gadap Town 8 3 11 452 463 Kemari Town 2 1 3 182 185 Lyari Town 9 0 9 250 259 Orangi Town 2 2 4 165 169 Karachi (Total) 24 9 33 1,243 1,276 Total 4 292 103 399 13,532 13,931 * The contract for one EMO school in Khairpur was awarded under the second RFP. Sindh Community Mobilization Program – Midterm Performance Evaluation 108 TABLE 4: SAMPLE FRAME Districts Sample Frame (Only Matched Schools) CMP Non-CMP Total EMOs Neighboring Other Sub-total Dadu 50 11 61 51 112 Jacobabad 37 13 50 38 88 Kashmore 42 8 50 41 91 Khairpur 3 41 12 56 51 107 Larkana 27 11 38 38 76 Qambar Shahdadkot 36 18 54 48 102 Sukkur 1 35 11 47 42 89 Districts (Total) 4 268 84 356 309 665 Bin Qasim Town 3 2 5 3 8 Gadap Town 8 3 11 14 25 Kemari Town 2 1 3 1 4 Lyari Town 9 1 10 11 21 Orangi Town 2 2 4 3 7 Karachi (Total) 24 9 33 33 66 Total 4 292 93 389 341 730 TABLE 5: GROUP INTERVIEW SAMPLE Districts Sample CMP Non-CMP Total EMOs Neighboring Construction/ Other Sub-Total Dadu 3 1 4 1 5 Jacobabad Kashmore 1 1 2 2 Khairpur 4* 1 1 6 1 7 Larkana 1 1 1 Qambar Shahdadkot 3 3 3 Sukkur 1 1 1 3 1 4 Districts (Total) 5 6 8 19 3 22 Bin Qasim Town Gadap Town 1 1 1 Kemari Town 1 1 1 Lyari Town Orangi Town Karachi (Total) 0 1 1 2 0 2 Total 5 7 9 21 3 24 Sindh Community Mobilization Program – Midterm Performance Evaluation 109 Table 6 indicates key characteristics for sampled schools, including number of schools that are high performing or low performing in terms of change in enrollment, number of schools by sex, and number of schools by level.28 TABLE 6: DISTRIBUTION OF SAMPLED SCHOOLS Type of School High Performing: Enrollment Growth Rate in 80th Percentile or Above (2008/2009 – 2015/2016) Low Performing: Enrollment Growth Rate Below 80th Percentile (2008/2009 – 2015/2016) Total Boys Girls Mixed Boys Girls Mixed Primary Secondary Primary Secondary Primary Secondary Primary Secondary Primary Secondary Primary Secondary EMOs 1 1 3 5 Neighboring 3 1 1 1 1 7 Other CMP 2 1 1 3 2 9 Non-CMP 1 1 1 3 Total 6 1 1 1 1 1 1 4 5 0 0 3 24 28 To determine high/low performing schools, the evaluation team calculated the growth in enrollment from 2008/09 to 2015/16. Those schools in the top quartile of enrollment growth were considered high-performing. Of the 730 schools in the matched sample, 170 are considered high-performing. Sindh Community Mobilization Program – Midterm Performance Evaluation 110 Table 7 provides a complete list of sampled schools by location, arranged in alphabetical order by school name. TABLE 7: SAMPLED SCHOOLS BY LOCATION SEMIS Code School Name District Tehsil UC Name 408180118 Ahmad Sector L1 Karachi Gadap Town Maimerabad Surjani 418050131 Arain Sukkur Sukkur Arain 426020550 Gaddu Barrage H.S. Kashmore Kashmore Kashmore Kashmore Colony No.1 427040073 Ghazi Markhand Kambar Shahdadkot Warrah Gaji Khuhawar 418030070 Government New Modern High School Sukkur Sukkur Sukkur Mir Masoom Shah 402030494 Gul Muhamm Dadu K N Shah Mitho Babar 415080167 Haryah Khairpur Mirs Sobhodero Sagyoon 415010528 Jhajhal Regulator Khairpur Mirs Faiz Ganj Kot Lalu 415010524 Karoondi Khairpur Mirs Faiz Ganj Karoondi 427020068 Khairo Gadhi Qambar Shahdadkot Miro Khan Khabar 415030378 Khoro Goth Khairpur Mirs Khairpur Mirs Khanrpur 402030240 KN Shah Dadu K N Shah Khairpur Nathan Shah 402030497 Mado Dadu K N Shah Chhore Qamber 415030398 Nizamani Khairpur Mirs Khairpur Mirs Shah Latif 413030032 Phulpota-I Larkana Rato Dero Behman 418020392 Rohri Sukkur Rohri Rohri 402040132 Saeed Pur Dadu Mehar Khan-jo-Goth 427030210 Sanjar Bhatti Qambar Shahdadkot Shahdadkot Jamali 408010093 Shahani Mohalla Karachi Kemari Town Mauripur 415060810 Tando Mir Ali Khairpur Mirs Mirwah Tando Mir Ali 415030402 Tando Nazar Ali Khairpur Mirs Khairpur Mirs Khanrpur 426010275 Turab Ali Muhammadani Kashmore Kandhkot Malheer 402010609 Waleed Shaikh Dadu Dadu Mounder 418010234 Wali Dino Khuharro Sukkur Pano Akil Dadloi Sindh Community Mobilization Program – Midterm Performance Evaluation 111 Annex 8: Background on Sindh Education Sector and Sector Reforms This annex provides useful background information and additional resources on enrollment trends in Sindh, development of school management committees, and recent education sector reforms in Sindh (including the School Consolidation Policy and Sindh Public-Private Partnership Act). Status of Education in Sindh Pakistan’s Constitution guarantees the right to free and compulsory education until grade 10, a right reinforced by Article 25-A of the landmark 18th Amendment to the Constitution passed in 2010. To implement Article 25-A, in 2013 the provincial assembly of Sindh passed ‘The Sindh Right of Children to Free and Compulsory Education Act, 2013,’ guaranteeing free and compulsory education for all children between 5 and 16 years of age, irrespective of sex or race. Nonetheless, gross enrollment rates in Sindh have stagnated in recent years and lag behind national levels. According to the Pakistan Social and Living Measurement (PSLM) survey, the gross enrollment rate (GER) for primary schooling in Sindh dropped from 82% in 2010-11 to 79% in 2014-15 (dropping from 71% to 69% for girls), compared to a constant rate of 91% at the national level during that time (82% for girls). Similarly, the GER for middle schooling dropped in Sindh from 59% to 55% from 2010-11 to 2014-15 (dropping from 50% to 48% for girls), compared to a constant rate of 62% nationally (increasing from 54% to 56% for girls). At the metric level, the GER in Sindh increased from 50% in 2010-11 to 51% in 2014-15 (increasing from 42% to 44% for girls), compared to a national increase from 55% to 58% during that time (increase of 48% to 50% for girls). For additional information regarding the status of education in Sindh, see the following: 1. Pakistan Bureau of Statistics. “Pakistan Social and Living Measurement Survey 2014-15.” http://www.pbs.gov.pk/sites/default/files/pslm/publications/PSLM_2014-15_National-Provincial￾District_report.pdf. 2. Provincial Assembly of Sindh Notification. “The Sindh Right of Children to Free and Compulsory Education Act, 2013.” March 11, 2013. http://www.sindheducation.gov.pk/Contents/Menu/Sindh_Bill_RTFE.pdf. 3. Reform Support Unit (School Education and Literacy Department, Government of Sindh). http://rsu-sindh.gov.pk/home/index.php. 4. Sindh School Education and Literacy Department. “Sindh Education Sector Plan (2014-2018)”. http://www.sindheducation.gov.pk/Contents/Menu/Final%20SESP.pdf. Community Engagement through School Management Committees (SMCs) Participation of parents and community members in education processes through the formation and strengthening of SMCs has been tested and found helpful in improving access to schooling in Pakistan. SMCs were introduced in Sindh in the early 1990s. The first Sindh Education Reform Program (SERP), developed in 2006, highlighted SMCs as a significant component of the GoS’s strategy to strengthen the education system. Building on foundations and lessons learned from the SERP, the Sindh Education Sector Plan (2014-2018) aimed to improve quality of service delivery and increase access to schooling through governance, accountability, and system improvements. SMCs represented a key building block for improving governance and accountability at the school level. The RSU has notified SMCs for all government schools in Sindh. The utility of these committees has been highly dependent upon personal interest and awareness of the head teacher and presence of influential members of a community on the committee who can mobilize resources for a school. At the Sindh Community Mobilization Program – Midterm Performance Evaluation 112 same time, the presence of influential community members can result in elite capture of committee resources or lack of interest of those members whose children are enrolled in private schools rather than in the government school. For additional information regarding community mobilization and SMCs in Sindh, see the following: 1. School Education and Literacy Department, Government of Sindh. “School Management Committees.” http://www.sindheducation.gov.pk/educationReforms/smc.jsp. School Consolidation For additional information regarding school consolidation, see the following: 1. School Education and Literacy Department, Government of Sindh. “School Consolidation.” http://www.sindheducation.gov.pk/educationReforms/sc.jsp. 2. Reform Support Unit, Education and Literacy Department, Government of Sindh. “RSU/School Consolidation Policy/2011.” February 16, 2012. Public-Private Partnerships in the Education Sector Historically, a number of public-sector initiatives, such as Idara-e-Taleem-o-Agahi (ITA), Pakistan Centre for Philanthropy (PCP), and Sindh Education Foundation (SEF), have engaged the private sector in education service provision in Sindh. These initiatives have resulted in direct private sector involvement in school management, teachers’ capacity building, and provision of scholarships to students. Programs under these initiatives have included Adopt-a-School Program, Concession to Private Schools, Afternoon School System, and Tawana Pakistan. Other corporate donors engaged with the public sector in education include the Federation of Chambers of Commerce and Industry, Engro Chemicals, Pakistan Petroleum Limited, Shell Pakistan, Unilever Pakistan Limited, and Proctor and Gamble Pakistan. For additional information regarding PPPs in the education sector, see the following: 1. Pakistan Reading Project (USAID). “Assessment Report: Existing PPP for Education Framework, Policy and Regulations in Pakistan.” 2016. http://www.pakreading.org.pk/sites/default/files/publications/PPP%20Assesment%20Report￾Final.pdf. Sindh Community Mobilization Program – Midterm Performance Evaluation 113 Annex 9: Impact of CMP on Enrollment The evaluation employed a quasi-experimental interrupted time series approach to estimate the impact of CMP on enrollment. The interrupted time series approach is a strong quasi-experimental design that is appropriate when an intervention occurs at a specific point in time and time series data on the anticipated outcome exists before and after the date of the intervention. The analysis looks for a significant change in the level or growth rate of the outcome that occurs at or after the intervention.29 For this evaluation, the analysis looks for a change in the growth rate of enrollment in CMP schools following the start of CMP. While the interrupted time series design is relatively strong, it can be difficult to isolate the effects of the intervention (CMP) from other factors that potentially affect the outcome. Adding a comparison group to the design improves the design’s ability to control for other factors. Long time series also strengthen the approach. Data To implement the analysis, the evaluation team asked the Reform Support Unit (RSU) for Sindh Education Management Information System (SEMIS) data on all 409 CMP schools and a large random sample of non-CMP schools for eight academic years (2008-09 through 2015-16). The team received complete data (i.e., data for all eight years) on 369 CMP schools and 44,711 non-CMP schools (including 14,819 non-CMP schools in CMP districts) distributed across all districts in Sindh. Data was disaggregated by sex but not by class. Variables included: • The school’s unique SEMIS code; • Location (district, tehsil, and union council); • Indicator variables for participation in CMP or the Sindh Reading Program; • School name; • School level (primary, middle, elementary, secondary, and higher secondary); • Gender (boys, girls, mixed); • Status (functional, closed, closed temporarily, closed permanently, on paper, and without SNE); • Number of rooms and classrooms; • Number of teachers (male, female, and total); • Enrollment (boys, girls, and total); and • Facilities (boundary walls, water, washrooms, and electricity). Propensity Score Matching The evaluation team used propensity score matching to construct a group of non-PMP schools as a comparison group for the analysis. Propensity score matching finds the non-CMP school that most closely resembles each CMP school based on characteristics of the schools. The matching exercise considered the CMP status of the school (CMP or non-CMP), level of the school (elementary/middle, secondary, and higher secondary), whether the school owned or rented the building, gender (boys, girls, and mixed), facilities (boundary wall, electricity, wash rooms, and water), the student/teacher ratio, and the student/classroom ratio). The exercise matched 339 non-CMP schools to 391 CMP schools. The numbers are different because the same non-CMP school was sometimes the best match for more than one CMP school. 29 The shift in level or rate of change is not necessarily associated with the date of the intervention since it may take some time for the intervention to take effect. Sindh Community Mobilization Program – Midterm Performance Evaluation 114 Limitations The evaluation team was not able to obtain SEMIS data on enrollment by class. Instead, it obtained enrollment data by type of school, i.e., primary, middle, elementary, secondary, and higher secondary. Because school type is defined by the highest class that attends the school, only primary schools contain exclusively primary classes. The analysis could not, therefore, assess trends in enrollment by class and only the analysis of primary schools reflects exclusively primary enrollment, but does not capture primary enrollment in other than primary schools. • SEMIS enrollment data reflect a great deal of variability. Possible explanations include the effects of the 2010 floods in Sindh which destroyed many schools and changes in reporting protocols. The evaluation team examined independent estimates of enrollment from the Pakistan Social and Living Standards Measurement Survey and found trends, and variability, similar to the SEMIS data, which suggests that external factors that affected both datasets may be responsible for the variability. The interrupted time series approach used regression techniques to attenuate the variability in the data. • The short two-year post-CMP time series does not provide a strong post-CMP trend in enrollment and thus compromises the strength of the enrollment estimates attributable to CMP. • SBEP is an integrated set of activities, of which CMP is one, focused on improving access to education in Sindh. Since the SBEP activities are tightly integrated, it is not entirely possible to identify the separate effects of individual components on enrollment. The evaluation’s selection of matched CMP and non-CMP (but still SBEP) schools separated the effects of CMP from other SBEP activities to the extent possible. Analysis The analysis used STATA’s integrated time series analysis (ITSA) routine to estimate the effects of CMP/SBEP on enrollment. For a model with a treatment and control group, the routine estimates the equation: 𝑌𝑡 = 𝛽0 + 𝛽1𝑇𝑡 + 𝛽2𝑋𝑡 + 𝛽3𝑋𝑡𝑇𝑡 + 𝛽4𝑍 + 𝛽5𝑍𝑇𝑡 + 𝛽6𝑍𝑋𝑡 + 𝛽7𝑍𝑋𝑡𝑇𝑡 + 𝜀𝑡 where: 𝑌𝑡 is the outcome variable (enrollment), 𝑇𝑡 is the year measured from the first year of the time series (0-7), 𝑋𝑡 is an indicator variable for the intervention (0=pre-intervention, 1=post-intervention), 𝑍 is an indicator variable for CMP schools (0=non-CMP, 1=CMP). and 𝜀𝑡 is a random error term. Figure 5 illustrates the interpretation of coefficients in a generic model with both treatment and control groups. A statistically significant coefficient 𝛽6 would imply that the intervention has a detectable immediate effect, i.e., it shifts the intercept relative to the control group intercept. A statistically significant coefficient 𝛽7 would imply that the treatment and control groups have different growth trajectories after the intervention. Table 8 illustrates the output from the ITSA routine. The results show that total enrollment in CMP schools was higher than in non-CMP schools at the beginning of the time series, i.e., 𝛽4 is statistically significant, but that CMP and non-CMP schools grew at comparable rates during the pre-intervention Sindh Community Mobilization Program – Midterm Performance Evaluation 115 period, i.e., 𝛽5 is not statistically significant. The intervention had no immediate effect on the outcome, i.e., 𝛽6 is not statistically significant, but did affect growth in the outcome, i.e., 𝛽7 is statistically significant. The effect of CMP on enrollment in the second year of the intervention is 𝛽7 (the difference in the annual enrollment growth rate) multiplied by the number of years CMP has been in operation (2). FIGURE 5: INTERPRETATION OF COEFFICIENTS TABLE 8: ITSA REGRESSION OUTPUT Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 8,255.9 552.6 14.94 0.000 𝛽4𝑍 9,446.2 3,086.0 3.06 0.016 𝛽5𝑍𝑇𝑡 290.1 961.7 0.30 0.771 𝛽2𝑋𝑡 -15,067.53 2,640.0 -5.71 0.000 𝛽3𝑋𝑡𝑇𝑡 -9,220.4 630.8 -14.62 0.000 𝛽6𝑍𝑋𝑡 -1,282.7 4,007.6 -0.32 0.757 𝛽7𝑍𝑋𝑡𝑇𝑡 3,940.9 1,171.7 3.36 0.010 𝛽0 98,889.2 1,847.0 53.54 0.000 Results Figure 6 summarizes the estimated contribution of CMP to enrollment for boys and girls at various school levels. All results are statistically significant except for the increase in enrollment (boys and girls) in primary schools. Sindh Community Mobilization Program – Midterm Performance Evaluation 116 FIGURE 6: SUMMARY OF CMP CONTRIBUTION TO ENROLLMENT Source: Analysis of SEMIS data. The tables below show the interrupted time series regression results associated with the changes in enrollment illustrated in Figure 5. Sindh Community Mobilization Program – Midterm Performance Evaluation 117 TABLE 9: GIRLS’ ENROLLMENT IN PRIMARY SCHOOLS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 612.3 172.4 3.55 0.012 𝛽4𝑍 3,078.0 403.6 7.63 0.000 𝛽5𝑍𝑇𝑡 97.0 205.8 0.47 0.654 𝛽2𝑋𝑡 -1,501.5 667.0 -2.25 0.065 𝛽3𝑋𝑡𝑇𝑡 -681.3 204.0 -3.34 0.016 𝛽6𝑍𝑋𝑡 562.3 721.8 0.78 0.466 𝛽7𝑍𝑋𝑡𝑇𝑡 -342.0 236.8 -1.44 0.199 𝛽0 11,839.3 256.3 46.19 0.000 Source: Analysis of SEMIS data. TABLE 10: BOYS’ ENROLLMENT IN PRIMARY SCHOOLS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 -115.9 171.4 -0.68 0.518 𝛽4𝑍 7,534.6 546.0 13.80 0.000 𝛽5𝑍𝑇𝑡 51.6 182.0 0.28 0.784 𝛽2𝑋𝑡 -1,413.1 519.7 -2.72 0.026 𝛽3𝑋𝑡𝑇𝑡 0.4 172.6 0.00 0.998 𝛽6𝑍𝑋𝑡 888.2 1,037.1 0.86 0.417 𝛽7𝑍𝑋𝑡𝑇𝑡 336.9 595.5 0.57 0.587 𝛽0 18,954.4 502.2 37.74 0.000 Source: Analysis of SEMIS data. TABLE 11: GIRLS’ ENROLLMENT IN HIGHER THAN PRIMARY SCHOOLS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 3,294.8 313.5 10.51 0.000 𝛽4𝑍 -2,688.2 549.9 -4.89 0.001 𝛽5𝑍𝑇𝑡 488.8 334.5 1.46 0.182 𝛽2𝑋𝑡 -5,171.3 1,774.5 -2.91 0.019 𝛽3𝑋𝑡𝑇𝑡 -1,902.8 740.3 -2.57 0.033 𝛽6𝑍𝑋𝑡 -2,100.1 1,986.5 -1.06 0.321 𝛽7𝑍𝑋𝑡𝑇𝑡 1,921.2 902.8 2.13 0.066 𝛽0 17,798.0 402.4 44.23 0.000 Source: Analysis of SEMIS data. Sindh Community Mobilization Program – Midterm Performance Evaluation 118 TABLE 12: BOYS’ ENROLLMENT IN HIGHER THAN PRIMARY SCHOOLS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 3,831.2 218.7 17.52 0.000 𝛽4𝑍 -979.8 1,868.4 -0.52 0.614 𝛽5𝑍𝑇𝑡 100.8 569.4 0.18 0.864 𝛽2𝑋𝑡 -5,301.5 1,165.8 -4.55 0.002 𝛽3𝑋𝑡𝑇𝑡 -2,529.7 503.5 -5.02 0.001 𝛽6𝑍𝑋𝑡 60.3 2,243.3 0.03 0.979 𝛽7𝑍𝑋𝑡𝑇𝑡 1,500.7 805.2 1.86 0.099 𝛽0 40,554.0 953.5 42.53 0.000 Source: Analysis of SEMIS data. TABLE 13: GIRLS’ TOTAL ENROLLMENT IN ALL LEVELS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 3,987.7 301.2 13.24 0.000 𝛽4𝑍 1,168.6 937.9 1.25 0.248 𝛽5𝑍𝑇𝑡 445.8 407.7 1.09 0.306 𝛽2𝑋𝑡 -7,343.9 1,491.7 -4.92 0.001 𝛽3𝑋𝑡𝑇𝑡 -5,684.2 356.7 -15.93 0.000 𝛽6𝑍𝑋𝑡 -2,095.9 1,790.5 -1.17 0.275 𝛽7𝑍𝑋𝑡𝑇𝑡 2,010.2 450.6 4.46 0.002 𝛽0 34,571.2 506.1 68.31 0.000 Source: Analysis of SEMIS data. TABLE 14: BOYS’ TOTAL ENROLLMENT IN ALL LEVELS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 3,747.6 272.7 13.74 0.000 𝛽4𝑍 6,604.8 1,944.4 3.4 0.009 𝛽5𝑍𝑇𝑡 102.5 562.1 0.18 0.860 𝛽2𝑋𝑡 -6,849.0 1,406.5 -4.87 0.001 𝛽3𝑋𝑡𝑇𝑡 -2,805.1 526.9 -5.32 0.001 𝛽6𝑍𝑋𝑡 1,042.7 2,664.3 0.39 0.706 𝛽7𝑍𝑋𝑡𝑇𝑡 1,891.5 1,099.7 1.72 0.124 𝛽0 59,985.8 1,101.3 54.47 0.000 Source: Analysis of SEMIS data. Sindh Community Mobilization Program – Midterm Performance Evaluation 119 TABLE 15: TOTAL ENROLLMENT, BOYS AND GIRLS AT ALL LEVELS Model Coefficient Estimated Coefficient Standard Error t P>|t| 𝛽1𝑇𝑡 3,747.6 272.7 13.74 0.000 𝛽4𝑍 6,604.8 1,944.4 3.4 0.009 𝛽5𝑍𝑇𝑡 102.5 562.1 0.18 0.860 𝛽2𝑋𝑡 -6,849.0 1,406.5 -4.87 0.001 𝛽3𝑋𝑡𝑇𝑡 -2,805.1 526.9 -5.32 0.001 𝛽6𝑍𝑋𝑡 1,042.7 2,664.3 0.39 0.706 𝛽7𝑍𝑋𝑡𝑇𝑡 1,891.5 1,099.7 1.72 0.124 𝛽0 59,985.8 1,101.3 54.47 0.000 Source: Analysis of SEMIS data. Sindh Community Mobilization Program – Midterm Performance Evaluation 120 Annex 10: SMC Subcommittee Membership, by Sex TABLE 16: SMC SUBCOMMITTEE MEMBERSHIP, BY SEX District Number of SMC Subcommittees Total Men Total Women Total Members Dadu 36 228 85 313 Jacobabad 38 228 81 309 Karachi 31 164 107 271 Kashmore 47 385 30 415 Khairpur 28 202 50 252 Larkana 43 275 111 386 Qambar Shahdadkot 34 222 82 304 Sukkur 25 144 78 222 Total 282 1,848 624 2,472 Source: CMP Annual Report, 2016. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523